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Abstract 

We consider a general class of discrete, two-dimensional Markov 
chains modeling the dynamics of a population with two types, without 
mutation or immigration, and neutral in the sense that type has no 
influence on each individual's birth or death parameters. We prove 
that all the eigenvectors of the corresponding transition matrix or in- 
finitesimal generator H can be expressed as the product of "universal" 
polynomials of two variables, depending on each type's size but not on 
the specific transitions of the dynamics, and functions depending only 
on the total population size. These eigenvectors appear to be Dirichlet 
eigenvectors for 11 on the complement of triangular subdomains, and 
as a consequence the corresponding eigenvalues are ordered in a spe- 
cific way. As an application, we study the quasistationary behavior of 
finite, nearly neutral, two-dimensional Markov chains, absorbed in the 
sense that is an absorbing state for each component of the process. 
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1 Introduction 



This paper studies spectral properties of two-dimensional discrete Markov 
processes in continuous and discrete time, having the neutrality property, in 
the sense of population genetics (see e.g. [17]). Considering two populations 
in interaction, corresponding to two different types of individuals (typically a 
mutant type and a resident type), one says that the types are neutral (or the 
mutant type is neutral w.r.t. the resident type, or more simply the mutation is 
neutral) if individuals of both types are indistinguishable in terms of the total 
population dynamics. In other words, the mutant population has no selective 
advantage (or disadvantage) with respect to the rest of the population. 

More formally, we say that a two-dimensional Markov process (X„, Yn)neT 
(where T = Z+ := {0, 1, . . .} or M+) with values in or is neutral if 
Zn = Xn + y„ is a Markov process. In particular, the law of the process Z 
depends on Zq, but not on Xq or Y^. 

If the process Z is a birth and death continuous-time chain, the class 
of neutral processes we consider is the following: the birth and death rates 
of the Markov process {Zt)tm.+ when Z is in state > are of the form 
k\k and k^ki respectively. Note that is an absorbing state for Z. With 
this notation, the parameters and /i^ can be interpreted as birth and 
death rates per individual. Then the process (X^, l^f)tgK_,_ is the birth and 
death process where both types of individuals have birth and death rates per 
individual and /ifc, when the total population has k individuals. This leads 
to the following transition rates for the Markov process {Xt.,Yt)t&M.+ : for all 



Note that the sets {0} x Z+, Z+ x {0} and {(0,0)} are absorbing for this 
process. In other words, we only consider neutral two-dimensional processes 
without mutation and immigration. 

In the case of discrete time, we consider two-dimensional birth and death 
processes constructed in a similar way: assume that the birth and death prob- 
abilities of the process {Zn)n&+ when in state k are and g^, respectively, 
with Pk + Qk ^ 1- Then, when a birth or a death occurs in the population, 
the individual to which this event applies is chosen uniformly at random in 



from to {i + l,j) 
from (i,j) to (i, j + 1) 
from (i, j) to {i - 1, j) 
from (i, j) to {i,j - 1) 



with rate 
with rate 
with rate 
with rate 




3 f^i+j- 
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the population. This leads to the transition probabilities 



from to {i + 1, j) 
from to + 1) 
from (ij) to {i - l,j) 
from (i, j) to - 1) 
from to 



with probability 
with probability 
with probability 
with probability 
with probability 



i+j p^+^ 
i+j ^*+-?' 

-J— n- ■ 

i+j Hi+3 



where := 1 — pk — Qk- Note that this construction requires assuming that 
Tq = 1 (i.e. that is absorbing for Z). 

In [2], Karlin and McGregor studied two families of neutral multitype 
population processes (branching processes and Moran model), but only in the 
case of nonzero mutation or immigration, for which the set of states where 
one population (or more) is extinct is not absorbing. They could express 
the eigenvectors of the corresponding infinitesimal generators in terms of 
Hahn polynomials. We focus here on neutral processes without mutation 
and immigration, which are singular for the approach of fH], and we apply 
our study to a much bigger class of neutral population processes, containing 
the birth and death processes described above, but also non-birth and death 
models. 

Our main result is the characterization of all eigenvalues and right eigen- 
vectors of the transition matrix of neutral processes without mutation and 
immigration. To this aim, we first consider the (easier) continuous state 
space case in Section [2] to introduce some tools used in the sequel. Next, we 
construct a particular family of polynomials of two variables in Section [3l 
using linear algebra arguments. In Section HJ we prove that the eigenvectors 
of the transition matrix of neutral two-dimensional Markov processes can 
be decomposed as the product of "universal" polynomials (in the sense that 
they do not depend on the specific transition rates of the Markov chain) with 
functions depending only on the total population size. We then relate these 
eigenvectors with Dirichlet eigenvalue problems in subdomains of of the 
form {(i,j) eN^ : i + j > k} ioT k >2 (Section E]), where N = {1,2, . . .}. 

The last section (Section [H]) is devoted to the application of the previous 
results to the study of quasi-stationary distributions. A probability distribu- 
tion u on \ {0} is called quasi- stationary if it is invariant conditionally on 
the non-extinction of the whole population, i.e. if 



P,((Xi,Fi) = (^,j)l^i^O) 



V(*,j)GZ+\{0}, 
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where P,^ denotes the law of the process (X, Y) with initial distribution u. 
This question is related to the notion of quasi-limiting distribution (also called 
"Yaglom limit", in reference to Yaglom's theorem on the same convergence 
for Galton- Watson processes), defined as 

I/,,, := lim P((X„,F„) = I Zn ^ 0), V(2,j) G Zl \ {0}. 

n— >+oo 

These notions are relevant in cases where extinction occurs almost surely in 
finite time, to describe the "stationary behaviour" of the process before ex- 
tinction when the extinction time is large. This is typically the case in many 
population dynamics models, where ecological interactions in the population 
produce high mortality only when the population size is large (one speaks of 
density-dependent models, see e.g. [18j, or [5j for discrete stochastic models). 

These questions have been extensively studied in the case where the tran- 
sition matrix restricted to the non-extinct states is irreducible (which is not 
true in our two-dimensional case). The first paper of Darroch and Seneta [6] 
studies the discrete-time, finite case. Several extensions of these results to 
continuous-time and/or infinite denumerable state spaces have then been 
considered in [2T| [71 [10]. The case of population dynamics in dimension 1 
have been studied by many authors(e.g. d [22l [161 [9l [13 [131 [HI [12]) -More 
recently, the quasi-stationary behaviour of one- dimensional diffusion models 
has been studied in |[2j. As far as we know, the two-dimensional case has only 
been studied in the continuous state space (diffusion) case [3j. An extensive 
bibliography on quasi-stationary distributions can be found in [20] 

In Subsection 16. H we first give the quasi-limiting distribution for gen- 
eral finite two-dimensional Markov chains in terms of the maximal Dirichlet 
eigenvalues of the transition matrix in several subdomains. Finally, in Sub- 
section 16. 2[ we apply our previous results to prove that coexistence in the 
quasi-limiting distribution is impossible for two-dimensional finite Markov 
chains which are close to neutrality. 

The paper ends with a glossary of all the notation used in Sections [H to [6l 
which may appear at different places in the paper. 

2 Preliminary: continuous case 

In this section, we consider the continuous state space case, where computa- 
tions are easier, in order to introduce some of the tools needed in the discrete 
case. 
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Fix p and q two measurable functions from ]R_|_ to ]R_|_, and consider the 
system of stochastic differential equations 

dXt = V^XtpiXt + Yt)dBl + XtqiXt + Yt)dt (la) 



dYt = ^2Ytp{Xt + Yt)dBl + Ytq{Xt + Yt)dt, (lb) 

where [3^,3"^) is a standard two-dimensional Brownian motion. Such SDEs 
are sometimes called branching diffusions, and are biologically relevant ex- 
tensions of the classical Feller diffusion |2]. If p and q satisfy appropriate 
growth and regularity assumptions, the solution to this system of SDEs is 
defined for all positive times and can be obtained as scaling limits of two- 
dimensional birth and death processes (we refer to [2j for the one- dimensional 
case; the extension to higher dimensions is easy). 

This process is neutral in the sense defined in the introduction since Zt = 
Xt + Yt solves the SDK 

dZt = Ztq{Zt)dt + ^2Ztp{Zt)dBt, (2) 

where 

is a standard Brownian motion. Note also that M+ x {0}, {0} x M_(_ and 
{(0,0)} are absorbing states as soon as there is uniqueness in law for the 
system ([1]). 

For any if G C^(]R^), the infinitesimal generator A of the process (X^, Yt)t>Q 
is given by 

A^{x, y) = (^x^{x, y) + V^i^, v)^ v{x + y) 

+ (x^{x,y) + y^{x,y)) q{x + y). (3) 



dx dy 

We first observe in the following proposition that A admits a symmetric 
measure, but only on a subset of C^(M^). 

Proposition 2.1 Assume that 1/p and q/p belong to L|^3^((0, +oo)). Let us 
define the measure fj, on as 

/i(ax, dy) = ^ — — — dx dy. (4) 

X y p{x + y) 
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Then, the restriction A of the operator A to C^((0, +00)^) is symmetric for 
the canonical inner product (■, in L^(]R^,/i), and, hence, so is its closure 
m L2(R^,/i). 



Note that, because of the singularities in fx when x or y vanish, if p > c > 
in the neighborhood of 0, any continuous function in L^(M^,/i) must vanish 
at the boundary of M^. Therefore, L2(R^,/i) c Lf^°(M^), where Lfo°(M+) is 

-■loci 



defined as the closure of Cc((0, +00)^) in hi ^^"^ 



Proof For all f,g G C^((0, +00)^), we have (formally) 



(/, Ag) 



j ^ p{x + y) (^^{x, y)^ix, y) + ^|~(^' ^) 

H ^ {x,y) — {x,y)] dx dy 

ox ox oy oy J 



+ j ^ fix, y)q{x + y) y) + y)j /"(x, y) dx dy. 
Therefore, (/, Ag)^ = {Af, g)^ if 

xq{x + y)fi{x,y) = Vx,?/>0 

and 

oivuv) 

yq{x + y)i2{x,y) = —^—{x,y), Vx,y > 0. 

Conversely, these equalities can be directly checked from the formula 
which implies that d{xfip)/dx and d{yixp)/dy exist in a weak sense. □ 

Before studying the eigenvectors of A, we need the following result. 

Proposition 2.2 For all A G M, the problem 

(1 - x^)h"{x) = -Xh{x) (5) 

has no (weak) non-zero solution h G C^([— 1, 1]) except when A = d{d — l) for 
some d eN. For d = 1, the vector space of solutions has dimension 2 and is 
spanned by the two polynomials h{x) = 1 and h{x) = x. For all d> 2, 

(l-x^)h"{x) + d{d-l)h{x) =0 (6) 
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has a one- dimensional vector space of solutions in C^([— 1, 1]), spanned by a 
polynomial of degree d, which can he chosen such that the family {Hd)d>2 
is an orthonormal basis of iJ^d— 1,1], j^^) . In addition, Hd has the same 
parity as d ( all powers of x that appear are even if d is even and odd if d is 
odd). 



Proof The case A = is trivial, so let us assume that A 7^ 0. Note that, 
assuming A 7^ implies that any solution h G 1, 1]) satisfying ^ 

must satisfy h{—l) = h{l) = 0. Indeed, the function h{x)/{l — x^) has a 
finite integral over [—1, 1] in that case only. The equation has the form of the 
classical Sturm-Liouville problem, but does not satisfy the usual integrability 
conditions, since (1 — is not integrable on [—1, 1]. 

First, uniqueness in C^([— 1, 1]) for ([5]) can be proved using the classical 
Wronskian determinant: given two solutions hi and /i2 of in C^{[—1, 1]), 
set W{x) := hi{x)h'2{x) - h[{x)h2ix). Since hi{-l) = hi{l) = for z = 1,2, 
we have W{—1) = W{1) = 0. Moreover, one has W'{x) = a.e. in [—1, 1] 
and thus W{x) = on [-1,1]. Thus {hi{0), h[{0)) and (/i2(0), /i^(0)) are 
linearly dependent, and so hi and /i2 as well. 

Second, assuming that has a polynomial solution of degree d, the 
comparison of the higher degree terms yields A = d{d — 1). So assume that 
A = d{d — 1) with d > 2. The polynomial 

h{x) = ^ ad,kx'', (7) 

0<k<d, d—k even 

where 7^ and 

{k{k-l)-d{d-l))ad,k = {k + 2){k + l)ad,k+2, VO < A; < (^-2, d-k even, 

is a nonzero solution of (|6]). Since h{l) = h{—l) = 0, we can choose ad^d 
to define the polynomial Hd '■= h satisfying ||i?d||p = 1, where p{dx) = 
(1 —x'^)~^dx and || ■ ||p is the norm corresponding to the inner product (-, ■)p. 

Third, let h G C^([— 1,1]) be a solution to for some A 7^ 0. Then 
h{x)/{l — x^) is continuous on [—1, 1], and for any d>2, 

1 



d{d — 1) J_i 1 — X 



.2' 



7 



where the integration by parts is justified by the Dirichlet conditions for h 
and Hii. Therefore, either A = d{d — 1) for some d > 2, and then / and 
are hnearly dependent by the previous uniqueness resuh, or 

{h,H,)p = f h{x)^^ dx = 0, Vrf > 2, 

where — x'^)}d>2 is a family of polynomials of all nonnegative 

degrees. Therefore, J -^ h{x)P{x)dx = for all polynomial P, and hence 
h = 0. 

The fact that the polynomials {Hd}d>2 are orthogonal w.r.t. the inner 
product (■, ■)p follows from ([8]) with h = Hd> for d' ^ d. 

Finally, assume that h G L^([— 1, 1], p) satisfies {h, Hd)p = for all d > 2. 
As above, we then have 




h{x)P{x) dx = 



for all polynomial P. Since h G L^([— 1, 1], p) C L^([— 1, 1], dx), such relations 
imply that h = a.e., which ends the proof that {-f^(i}d>2 is an orthonormal 
basis of L^([— 1, 1], p). The parity property of follows from ([7]). □ 



Remark 2.3 The polynomials may be obtained as a limit case of Gegen- 
bauer polynomials, defined for a parameter A > — 1/2 as (up to a multiplica- 
tive constant) 



for all d > Q, where B is the Beta function, aPb o-f^^ the classical hypergeo- 
metric series and {x)n '■= x{x + 1) . . . (x + n — l) is the shifted factorial. The 
normalizing constant has been chosen such that these polynomials form an 
orthonormal family in L^((— 1, 1), (1 — x'^)'^^^^'^dx) . Then, for all d > 2, 
may be obtained as 



2d -I j^[-d)t,{d-i)u (i + x; 
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One easily checks from this formula that if^ solves and the fact that 
J^^Hd{xy{l — x'^)^^dx = 1 is obtained applying the limit X — —1/2 in 
j\H^{xf{l - x^y-^'^dx = I. 

We now introduce the change of variables 

z = x + ?/eR+ and w = ^ G [-1, 11, (10) 

X + y 

which defines a C°°-diffeomorphism from M^\{0} to (0, +oo) x [—1, 1]. Then 
^{dx,dy) = u^dzjdw), where 



2exp {irM^^ 



vidz, dw) := ——^ — ^. , . dz dw, 

z[l — w^)p[z) 

and 

A = ^P(^)q-2 + + -^W^- (11) 

Now, assume that A has an eigenvector of the form ip{w)il){z). If A is the 
corresponding eigenvalue, (fTTl) yields 

^{w) [zp{zW{z) + zq{z)ij'{z) - A^(^)) + (1 - w^)^"{w)^i){z) = 0. 

Hence, we have the following result. 
Proposition 2.4 All functions on of the form 

HJ^^)i^{x + y\ (12) 
\x + y/ 

where d>Q, Hi{x) = x, Hq{x) = 1, H^, k > 2 are as in Proposition \2.Bf 
where ip satisfies 

zp{z)^"{z) + zq{z)^'{z) - d{d - l)^tlj{z) = Xtlj{z), "iz > 0, (13) 

z 

for some A G M, are eigenfunctions of A for the eigenvalue A. 
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Now, we proceed to prove that there are no other eigenvectors of A (in 
the sense of Theorem 12.51 below). Such a result seems natural as f ll3p can be 
written as the Sturm-Liouville problem 

T. (^•'(^iexp/'||.,)-.(.-l)!!i4|i^,(.) ^ -A^^5^V-(.). 

Here again, the usual integrability conditions are not satisfied. More pre- 
cisely, if p{z) > c > for z in the neighborhood of and q is bounded, 
using the terminology of j23], for all d > 0, the problem ( fT2l) is a singular 
self-adjoint boundary value problem on (0,oo), where the endpoint is LP 
(limit point) singular (see p3| Thm. 7.4.1]). In this case very little is known 
about the existence of an orthonormal basis of eigenvectors in L^((0, oo),u), 
where 

(the spectrum might even be continuous, see [23, Thm. 10.12.1(8)]). 

For this reason, we state our next result on the operator A with a re- 
stricted domain corresponding to the case where the diffusion is reflected in 
the set 

D := {{x,y) e Rl : a < X + y < b}, 

where < a < 6 < oo. For all initial condition {Xq,Yq) = (a;o,yo) ^ D, we 
consider the process {Xt,Yt,kl"'\kf'^)t>Q such that k^^^ = k^^ = 0, A;'-"^ and 
are nondecreasing processes, {Xt, Yt) G D for all t > 0, 

k^r^ = [ lix.+Y.=a}dki^\ Vt > 0, a = a,b 
Jo 

dXt = V2Xtp{Xt + Yt)dBl + Xtq{Xt + Yt)dt - ^'^dkf^ + v^'^dki""^ 
dYt = ^/2Ytp{Xt + Yt)dB^ + Ytq{Xt + Yt)dt - V^T^dkf^ + y/Y^dkl^K 

Then Zt = Xt + Yt is the solution of (|2]) reflected at a and b with local time 
k'f'' at a (resp. kf^ at b). Therefore, {XtiYt) is also neutral in the sense of 
the introduction. 

The corresponding infinitesimal generator is defined by ([3]) with domain 
the set of if{x,y) G C^{D) fl C'^{mt{D)), where int(D) denotes the interior 
of D, such that 

dip dip 

— (x, y) + "^(^' y) = 0' V(a;, y) e D s.t. x + y = a oi b. 
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Theorem 2.5 For < a < b < oo, assume that p > c > on [a,b] and q/p 
belong to L^([a, b]). 

(a) There exists a denumerable orthonormal basis oflJ^{D,fi) of eigenvec- 

tors of A of the form [T^) . where d > 2 and ip solves (Q^j on (a, b) 
and satisfies ip'{a) = il)'{b) = 0. Moreover, any eigenvector of A in 
lJ^{D,n) is a linear combination of eigenvectors of the form l[T^) . all 
corresponding to the same eigenvalue. 

(b) There exists a family of right eigenvectors of A of the form 

Ix + y X + y J k>i 



[j{Hj^y^\x + y)} , (15) 
^ I \x + y/ J k>i 

d>2 ^ 



which is a basis of the vector space 

V ■.= \j ^ L2(D,Leb) : 3/i,/2 e L2([a, 6], Leb) and /s G l^iD.y^) s.t. 

fix, y) = -^fi{x + y) + -^f2{x + y) + f,{x,y)], (16) 
X + y X + y J 

where Leb denotes Lebesgue's measure. More precisely, for all f G 
V, the functions /i,/2,/3 in / fT^) are unique and there exists unique 
sequences {ak}k>i, {Pk}k>i, and {'ydk}d>2, k>i such that 



fix, y) = Y] ak—^ipk{x + y) + Y] (3k—^ipk{x + y) 
^-^ X + y ^-^ X + y 

k>l ^ k>l ^ 

+ E l^^H,[^)i^f\x + y), (17) 

d>2 k>l ^ 

where the series Ylk'^k'^Pk and Ylkf^k'^k both converge for \\ ■ \\^ and 
T.d,k^dkHd{^)ipf\x + y) converges for \\ ■ \\^. 

Point (b) says that the eigenvectors of the form (fT2l) . ahhough not orthog- 
onal in some Hilbert space, aUow one to recover a bigger class of functions 
than in Point (a). The vector space V is not equal to L^(D, Leb), but the 
following result shows that it is much bigger than L^(D,/i). 

Proposition 2.6 The vector space V of Theorem \2.5\ contains H^{D,Leh) . 
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Proof To prove this result, it is more convenient to consider the variables 
{z,w) as in (ITU]) instead of {x,y). The vector space V then becomes the set 
oi g e L^([a, b] x [—1, 1], Leb) such that 

1 + w 1 — w , , 

g{z, = -^—gi{z) + -^—g2{z) + g^[z, w) (i8) 

for some (71,(72 G L^([a, 6], Leb) and (73 G L^([a, 6] x [— l,l],z/) = L^([a,6] x 
-w'^y^dzdw). 

We first introduce the following notion of trace: we say that a function 
g G L^([a, h] x [—1, 1], Leb) admits the function g G L^([a, 6], Leb) as a trace 
atw = l, orw = — 1 respectively, if 

g{z,w) - g{z) G h^{[a,b] x [0, 1], (1 - vo^Y^dzdw), 

or 

g{z,w)-g{z) GL^([a,6] x - w'^Y^dz dw) 

respectively. 

Our first claim is that any g G L^([a, h] x [—1, 1], Leb) which admits traces 
gi and (72 at w = 1 and w = —1 respectively, belongs to V , and these traces 
are exactly the functions gi and (72 in (H^ . To see that, we only have to 
check that g^ G L^([a, 6] x [0, 1], (1 — w'^)~^dzdw)^ and the same result on 
[a, 6] X [—1,0] will follow by symmetry: 



<2 /" / dw ^^~^^^ +2 / rfz / du; ((7i + (72)^7^7^^ < +00. 



a ^0 



1 - ^ Ja Jo " M(l + u;) 



Second, we claim that any g G H^{[a,b] x [—1, l],Leb) admits traces at 
w = 1 and w = — 1 as defined above. Assume first that g G C^{[a, b] x [0, 1]). 
Then, using the Cauchy-Schwartz inequality, 

Q Ja l-W^ Jo J a 1 - W^ 

1 rb rl 







< dw dz dx \'Vwg{z,x)\ < ||V^(7| 



Leb- 
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Since in addition l)||L2(Leb) ^ 4||5f||j:/i(Leb) by classical trace results (cf. 
e.g. [H p. 196]), the function g i— )■ ((?(■, 1), g — (?(■, 1)) extends by density to a 
linear operator ip continuous from if^([a, 6] x [0,1], Leb) to L^([a, 6], Leb) x 
L^([a,6] X [0, 1], (1 — w'^)~^dzdw). Since obviously ipi^g) + '>p2{,g) = 9, the 
claim is proved and the proof of Proposition 12.61 is completed. □ 



Proof of Theorem 12.51 An eigenvector of A of the form Hd{w)il){z) satis- 
fies the Neumann boundary condition in D iff il}'{a) = ip'{b) = 0. The prob- 
lem (fT3|) with this boundary condition is a regular Sturm-Liouville problem 
with the weight u defined in (1141) . Therefore (cf. e.g. Thm. 4.6.2]), for all 

d > 0, there exists an orthonormal basis {'ip^^}k>i of ^'^{[O'^b],'^) composed 
of solutions to (I13p on (a, b) with Neumann boundary conditions. All the 
corresponding eigenvalues are real, simple and the corresponding spectrum 
has no accumulation point. 
Now, we claim that 



VI \x + y/ ) k>i 



d>2 



forms an orthonormal basis of L^(D,yu). The orthonormal property fol- 

' x—y ^ 
^x+y^ 

HAI:^y)^^^\x + y) for d,d'>2 and k, k' > 1, 



lows from the fact that, if if{x,y) = Hd{^rr')ipl^\x + y) and (f'{x,y) 



(<^,<^%= / / Hd{w)Hdiw)4'\z)^l^'\z)du{z,w) 

J-l J a 

To prove that J-" is a basis of L^(D,/i), assume that / G L^(D,/i) satisfies 

// f{x,y)Hd{'^)'^pf{x + y)d^^{x,y) = Q, W>2,k>l. 
J J D \x + y/ 

By the Cauchy-Schwartz inequality, for all d > 2, the function 
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belongs to L^([a, b],^). In addition, 

{fd,4% = 0^ VA;>1. 

Therefore, fd{z) = for all d > 2, for Lebesgue-almost every z > 0. By 
Fubini's theorem, w t-)- /(^^^y^, Mli:^) belongs to L^([— 1, 1], (1 — x'^)^^dx) 
for almost every z > 0. Hence we deduce from Proposition 12.21 that this 
function is for almost every z > 0. Hence / = 0. 

Thus J-" is an orthonormal basis of lJ^{D,fi) composed of eigenvectors of 
A. It is then classical to deduce that A admits no other eigenvector in this 
space, in the sense of point (a). 

For point (b), let us first prove that the decomposition 

X y 
f = — : — fi{x + y) + — ■ — f2{x + y) + f3ix,y) 
X + y X + y 

is unique for / G V^, with /i, /2 G L^([a, b], Leb) and /a G lJ^{D, /i). We only 
need to prove that this equality for / = implies /i = /2 = /s = 0. 
Since /s G L^(D,/i), we have 

fdx f \yfi{x,y)<s f dx f \y = oie) 

Jo Ja-x Jo Ja-x X 

as £ — )■ 0. Therefore, 

fd- P^^7^/.'= fdx rdy ij^fl + h)' 
Jo Ja-x yx + y) Jo Ja-x \X + y / 

This implies that f2{z)dz = 0, i.e. /2 = 0. Similarly, /i = and thus 
/s = 0. 

Since L^([a, 6], Leb) = L^([a, 6], z/), the result then follows from the de- 
composition of fi and /2 (resp. /s) in the orthonormal basis {'ipk^^}k>i of 
h\[a,b],u) (resp. {H,{^^)4'\x + y)},>2,k>i oih\D,fi)). ~ □ 

To motivate the calculations of the next section, let us finally observe 
that, for all cp G C^Rl), 

A(p{x, y) = f(p{x, y)p{x + y) + L(p{x, y)q{x + y). 
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where 



and 



and that TL = LT 

3 On a family of bivariate polynomials 

The goal of this section is to prove the existence of a family of polynomials 
in M of two variables X and Y , satisfying the family of relations 

XP{X + 1, y) + YP{X, y + 1) = (X + y + d)P{X, Y) (20a) 

XP{X -l,Y) + YP{X, Y -1) = {X + Y - d)P{X, Y) (20b) 

for an integer d > 0. 

Before stating the main result of the section, let us recall some notation. 
M[X] is the set of polynomials on IR with a single variable X and R[X, Y] 
the set of real polynomials with two variables X and Y. The degree deg(P) 
of a polynomial P G M[X, Y] is defined as the maximal total degree of each 
monomial of P. We define 

Vd^{PeR[X,Y]:degiP)<d}. 

For all P e M.[X, Y], we may write 

P(X,F) = 5^a,,XY^ 

i,j>0 

where only finitely many of the aij are nonzero. The real number will be 
called the (i, j)-coefficient of P. 

For any P e M[X, Y] and for any d > 0, we denote by [P]d the sum of all 
monomials of P of degree d: 

d 

[PUX,Y) = J2<^^,d-^X'y'~'■ 

i=0 



d_ 

dx 



d_ 
' dy 



T 



X 



dx' 



+ y 



(19) 
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In particular, [P]^ is homogeneous of degree d and P = Xli^il-^]*- 

We denote by Aj the first-order symmetrized discrete derivative with 
respect to the z-th variable: 

VP e nx, n A,p(x. y) = - ^'^ - 



and A2P{X, Y) 



2 

P{X,Y + 1)-P{X,Y-1) 



and by Af the symmetrized second-order discrete derivative with respect to 
the z-th variable: 



VPGM[X,y], AiP{X,Y) 
and AlP{X, Y) 



2 ^ P(x + 1, y) + P(x - 1, r) - 2P{x, Y) 



2 

p(x, r + 1) + p(x, r - 1) - 2P(x, F) 



Note that the superscript in the notation does not correspond to the 
composition of the operator Aj with itself. 

Finally, we define the linear operators on Y] 

L = XAi + YA2 and T = XAj + YAj. 

Then, adding and substracting the equations (I20ap and ( ]20bp . the system (pO!) 
is equivalent to 

LP = dP (21) 
TP = 0. (22) 

We are going to prove the following result 



Theorem 3.1 For d = 1, the system (120|) has a two-dimensional vector 
space of solutions in M[X, Y], spanned by the two polynomials Pi^^ := X and 
Pf :=F. 

For any d G {0,2,3, . . .}, the system f l20|) has a one-dimensional vector 
space of solutions. All these solutions, except 0, are of degree d. For d = 0, 
this is the vector space of constants, spanned by Pq := 1. When d > 2, we 
denote by P^ the unique solution to ( 120]) with [d — 1, 1) - coefficient equal to 
-2H'^{1) = {-iy2^yd{d - l){2d - 1) , where is defined m RemarklKE 
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It can be checked that the first polynomials are 

P2 = 2V6 XY, 

P3 = -2V3O XY{X -Y), 

P4 = AV2I XY{X^ - 3XY + Y^ + 1), 

P5 = -6V26 XY{X - Y){X^ - 5XY + Y^ + 5). 

Before proving this result, let us give some properties of the polynomials 
Pd, proved after the proof of Theorem 13. II 

Proposition 3.2 The polynomials Pd, d >2, defined in Theorem \3. 1\ satisfy 
the following properties: 

(a) For all d>2, [Pd]d{X, Y) = {X + YfHd (f^), where Hd is defined m 

RemarklKE 

(b) [P]d-2k-i = OforallO<k< d/2. 

(c) For all d > 2, Pd is divisible by XY . For d odd, Pd{X, Y) is divisible by 

XF(X -r). 

(d) for all d>2, Pd{Y,X) = Pd{-X, -Y) = (-1)'^P,(X, F). 

(e) Pd{i,j) =Oift,je Z, tj >0 andO< \t\ + \j\<d-l. 

(f ) For all d > 0, the matrix {Pi{j, d—j))o^i j<d is invertible, where Pi = P^^^ . 

In particular, {Pd{j, d - j))o<j<d ^ 0. 

(g) For all d>3, PdU, d - j)Pdij + 1, d - j - 1) < if 1 < j < d - 2. 

(h) For all d, d', k>2, 

^ Pd{i,k - i)Pd'{i,k - i) fk + d-l\ 

g — ^^37^ — =\2d-i y^'' 

where Sij is the Kronecker symbol and by convention Q) = if j < 
or j > i. 
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Proposition 3.3 For all d > 2, the polynomial is given by the following 
formula: 



P.ix.y^^c,-x-yu p-^^-:^\}-X ^ (24) 



(b) DeGne for aUd>0 

Vd = {PeVd: T{P) = 0}. 
Then dim{T>o) = 1 and dim{Vd) = d + 2 for all d > 1. 

Proof of Lemma 13.41 To prove ( !25|) . it would be enough to expand both 
sides of the equation. We prefer to give a proof based on differential cal- 
culations, because it is related to the method used in the rest of the proof. 
First, let / = {1, 3, 5, . . .} and J = {2, 4, 6, . . .} be the sets of odd and even 
integers, respectively. Using the fact that, for Q G ]R[X]. 



where 




In order to prove Theorem 13. we need the following lemma. 



Lemma 3.4 

(a) We have 



TL = LT + T, 



(25) 




and 




n>0 



n>0 



where Q*-"-* denotes the n-th derivative of Q, one has 




(27) 



(26) 
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Since for all p, g G N 



dXP V dXi J dXP+i ' 
one easily checks that 

TL-LT= ^ ^^(^-^)(^^X^ + ^9F^ 
Now, for all n eN, 

y y 11- y 11 

p&J,q&I,p+q=2n+l^' ^' p&I , qGl ,p+q=2n ^' ^' p€j, qeJU{0} , p+q=2n 



»!(2n-p)!' ' (2n)\ 



(2n)! (2n)!' 

This completes the proof of Lemma [3.41 (a). 

To prove (b), let "Hp be the subspace of Vd composed of all homogeneous 
polynomials of degree d. Recall the definition (fT9|) of the operator T on 
y], and observe that, for all d > 1, T is a linear map from T-id to T-id-i- 
Now, the family {(X - Yf^X + F)'^-^0 < k < d} forms a basis of Ud- 
Hence, any P G Hd can be written in the form P(X, F) = (X + YYh ( ^~y ) , 
where /i G M[X] has degree d. With this notation, it can be checked that 

^^{X,Y) = d{d-l){X + YY-'h{W) 

+ A{d - l)Y{X + + 4F2(x + YY~^h"{W), 

where W = {X — Y)/{X + F), and similarly for the second variable. This 
yields 

~ / XY 

TP={X + Yf-' [d{d - l)h{W) + 

Using the relation 4XF/(X + F)^ = 1 - T^^, we finally obtain that P G 
Ker(T) fl'Hrf if and only if h solves ([6]). By Proposition 12.21 for all d ^ 1, this 
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equation has a unique (up to a multliplicative constant) polynomial solution, 
which has degree d. Since dim('Hrf) = + 1, we deduce that T : 1-1^ — t- l-Ld-i 
is surjective for all d > 2. If d = 1, Ker(T) fl Hi = Tii which has dimension 
2. 

Now, let P = [P]i + . . . + [P]d G Vd and observe that any k-th order 
derivative of [P]m belongs to Hm-k if k < m. Therefore, by fl2B]) . the equation 
TP = is equivalent to the fact that, for all < n < — 1, 

[^^]" = ^ (2^ [^dX^P + ^91^ j [PU2P-1 = 0, 
or, equivalently, 

riP]„„ = -2 E (.f + y aps; j H-*- (28) 

If n > 1 and [P]„+3, [P]„+5, . . . are given, there is a one- dimensional affine 
space of solution for this equation. If n = 0, (128|) is automatically satisfied, 
since both sides are 0. Therefore, choosing recursively [P]d, [P]d-i, ■ ■ ■ , [P]2 
and setting any value to [P]i and [P]o, the result on the dimension of Vd 
easily follows. □ 



Proof of Theorem 13.11 Fix d > 0. We claim that, as a linear operator 
on Vd, L is diagonalizable and its spectrum Sp'p^(L) = {0,1, . . . ,d}. To see 
this, fix A G Spp^(L) and P an eigenvector for this eigenvalue, with degree 
p. Writing as in the proof of Lemma [3.41 P = [P]p + . . . + [P]o, the equation 
LP = XP is equivalent to the fact that, for < n < p 

= E [^W^. + ^aFSTTj \PU.,. (29) 

Now, for any Q G T-ik, one has 

LQ = + = kQ. (30) 

^ dX dY ^ ^ ' 

Therefore, (129|) for n = p imposes X = p, and for n = p — 1, [P]p-i = 0. 
Moreover, for < n < p — 2, (129|) is equivalent to 
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which allows one to compute recursively [-P]p-2, ■ ■ ■ , [P]o given any [P]p G Tip 
and [-P]p-i = 0. Since dim('Hp) = p+1, the eigenspace corresponding to the 
eigenvalue p of L has dimension p + 1. 

Now, it follows from (125|) that L is a linear operator from Vd to P^. Since 
L is diagonalizable in Vd, it is also diagonalizable on any stable subspace, 
and Spp^(L) C {0, . . .,d}. 

Let p E {2, . . . ,d} and assume that there exists P E Vd\ {0} satisfying 
LP = pP. Again, deg(P) = p necessarily. Writing P = [P]p + . . . + [P]o 
again, since T(P) = 0, we have 

which has a one-dimensional vector space of solutions in Tip. Once [P]p is 
fixed and since we have [P]p-i = 0, (13T]) can be used recursively to compute 
[P]p_2, . . . , [Pjo- In conclusion, the eigenspace of L in Vd for the eigenvalue 
p is either of dimension 1 or 0. Now, L is diagonalizable in Vd- Since 
dim(Pd) = d + 2 and dim('Ho) + dim('Hi) = 3, the only possibility is that 
Spj, (L) = {0,1, ... ,d} and that each eigenvalue p ^ 1 has a one- dimensional 
vector space of solutions, and the eigenvalue 1 has a two-dimensional vector 
space of solutions. 

This easily implies Theorem 13. except for the expression of —2H'^{1). 
This can be easily obtained from ([9]) and the relation H'^{1) = (— 1) 
which follows from the parity property of Hd stated in Proposition 12.21 □ 



Proof of Proposition 13.21 Recall from the proof of Lemma 13.41 that a 
nonzero solution P of ( 120|) for d >2 satisfies 

d 

n=0 

where h{x) = Yln=obnX"' is a polynomial solution of ([6]). Therefore, the 
{d — 1, l)-coefScient ad-i^i of P is given by 

d 

aa_i^-^ = hn{-n + d-n) = dh{l) - 2h'{l) = -2h'{l), 

n=0 
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and Point (a) then follows from Proposition 12. 21 and the value of the (c? — 1, 1) 
coefficient of P^. 

Observe that any polynomial solution of (|6]) with d > 2 is divisible by (X— 
+ 1). As a consequence of the previous construction, any polynomial P 
such that (120|) holds satisfies that XY divides [P]d- Note also that [-P]d-i = 0, 
which implies (b) by 0311) . Moreover, (1ST]) also implies by induction that XY 
divides [P]d-2k ioT all < < d/2, which yields the first part of (c). 

By Proposition |2l2l (d) is true for [Pd]d and of course for [Pj^-i = 0. Now, 
assume that P G R[X,Y] satisfies P(F,X) = P(-X, -Y) = (-l)"P(X,r). 
Then it can be easily checked that, for all /c > 1, 

gkp Qk p 

satisfies 

Q{Y, X) = (-l)"g(X, Y) and Q{-X, -Y) = (-l)"+^+ig(X, Y). 

Therefore, (d) easily follows from ( 13T]) by induction. 

Now, fix d odd. By Proposition 12. 2[ is odd and thus Ha{X) is divisible 
by X. This implies that [P]fi is divisible hy X — Y. Moreover, it follows 
from (d) that the polynomial [P]d/ (X— y) is symmetric. Now, let Q{X, Y) = 

^nyra ^ ^rayn g^^g n, m > 0, and fix > 0. SluCC 

dX^ dY^ V / V I 7v / 

+ m(m - 1) ... (m - A; + - x"F'"-'^+i), 

the polynomial X — Y divides X|^ — Y^^. Since this holds for any 
n, m > 0, the same is true for all Q such that Q{X,Y) = Q{Y,X). Now, 
any polynomial of the form P(X, Y) = (X — Y)Q{X, Y) with Q symmetric 
satisfies 

gkp Qkp d^^^O d'^^^O 

X^(X, Y) + Y^(X, Y) = kX^—^iX, Y) - kY^^—^iX, Y) 



In particular, X — Y divides X|^ + Y^^. Therefore, the fact that [P]j is 
divisible by X — F for z < follows form (13T]) by induction. This ends the 
proof of (c). 
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As a consequence of (c), Prf(i, 0) = Pd{0,j) = for any i,j & Z for d >2. 
Applying (|20B for (X, F) = (rf - 1, 1) yields Paid - 2, 1) = 0. By induction, 
applying fl20b|) for (X, F) = (d - fc, 1) implies Pd{d - A; - 1, 1) = for all 



k e {1, . . . , d - 2}. Similarly, applying fl20bD for (X, Y) = {d - 1 - k,2) 
implies Pd{d — 2 — k,2) =0 for all A; G {1, . . . , ci — 3}. Point (e) is therefore 
straightforward by induction. 

For all d < k, the polynomial Qk,d{X) = Pfc(X, d — X) satisfies [Qk,d]k = 
[d^Hk 7^ for all k > 2. Therefore, deg(gfc,d) = k for all k > 0, 

and {Qo,d, Qi,d, • • • , Qd,d} is a basis of := {Q e M[X] : deg((5) < d}. Since 
V'(Q) = (Q(0)) • • • 5 Q{d)) defines a linear isomorphism from V'^ to M''"'"^, we 
deduce that {'^{Qo d)-, ■ ■ ■ i^{Qdd)} is a basis of which is equivalent 

to (f). 

Point (g) is a simple consequence of points (e) and (f ) and of formula fl20ap 
with X = j and Y = d — j — 1. 

Because of point (e) above, (h) is obvious ii k < d — 1 oi k < d' — 1. 
So let us assume that rf, d' < k. Multiplying ( \20a\} by (X + Y — d + 1) and 
applying ( I20bl) to both terms on the l.h.s. yields 



(2XF-rf(rf- l))Prf(X, Y) = Xr(Prf(X + 1, Y - 1) +Pd(X - 1, F + 1)). (32) 



This means that, for all A; > 2 and 2 < d < k, the vector {Pdih k — i))i<i<k-i 

h,j )l<i,j<k-l 



is a right eigenvector of the matrix = (a i<j j<fc_i for the eigenvalue 



—d{d — 1), where 

af^^ = -2i{k -i)ioT l<i<k-l, 
af^^^ = i{k-i)ioi2<i<k-l, 
afl}_^ = i{k - i) ioT 1 < i < k - 2 
and a,-J^^ = for \i — j\ > 2. 

It is straightforward to check that the matrix A^. is self-adjoint for the inner 
product (-, where /ij = l/i{k — i), which implies that two right eigenvec- 
tors of Ak corresponding to different eigenvalues are orthogonal w.r.t. this 
inner product. This yields (|23|1 for d ^ d' . 
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Finally, fix 2 < < A;. Using fl^Oal) and fl20b|) . we have 
^ lik-i) 

i=l ^ ' 

= yPd{t,k-t + l) -— + yPd{t,k-t + l) : 

^-^ k — t + I ^-^ % 

i=\ 1=1 



Applying this equality inductively, we deduce that 



2d - 1 



for some constant C. 



Now, by point (a) and Proposition 12. 2[ we have 



k^'^~'^ ^ i{k-i) k^i_(^^Y J_^l-x^ 
as — )■ +CXD. Thus C = 2 and the proof of (h) is completed. □ 



Proof of Proposition 13.31 In [T3], the authors construct a family of func- 
tions of two variables satisfying relations close to ( 120|) . which they use to 
study neutral, multitype population processes with non-zero mutation or im- 
migration. These functions are expressed in terms of the Hahn polynomials, 
defined for fixed parameters «>— 1, /3>— 1 and G N by 

Q.(x; a, /3, AT) = j^"^' '"^^^^ +^ + ^ + ^ , (33) 

for all integer c? > 0. Karlin and McGregor proved that the rational function 

MX. Y) = Q4X, «, /3, X + r + 1) = ^ ia + lU-X-Y),kl 
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satisfies 



(X + a + l)MX + 1, r) + (F + /3 + 1)0,(X, Y + l) = {X + Y) 

(34) 

X04X-l,F) + F0d(X,F-l) 

{X + Y + l-d){X + Y + d + a + (5 + 2) ^ (35) 

x + r + 1 

Let us define 

7/>,(X, r) = (-X - r), lim lim (a + 1)0,(X, F) 
. ^ ^ (-rf),(-X),(d-l), 



-(fc-i)!(-x-r), 



Passing to the limit in and (135|) proves that ipd satisfies (120|) . Since t/^^ 
is a polynomial, Theorem 13.11 entails ( I24p . It only remains to check that 
Cdipdi^X.Y) has its (rf - 1, 1) coefficient equal to {-l)'^2^J d{d - l){2d - I). 
The (d — 1, 1) coefficient of ipdiX, Y) is 



- (A:-1)!A:! ^ ^ ^ ^ ^ 

2 

The Chu-Vandermonde formula (cf. e.g. [8j) implies that 



2 ' ; ' {d-l)\ 
which gives the expression of C^. □ 

4 Spectral decomposition of neutral two-dimensional 
Markov chains 

In this section, we consider neutral extensions of the two-dimensional birth 
and death chains in described in the introduction. In the sequel, this 
family will be called N2dMC, for neutral two-dimensional Markov chains. 
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A N2dMC {Xt,Yt) tez+ is constructed by specifying first the Markov dy- 
namics of Zf — Xt + Yf in Z+. Assume tliat its transition matrix is 

no = (Pn,m) 

wfiere X]m>oP«,m ~ ^ all n > 1 and the state is absorbing (po,o = !)• 
Then, the process {Xt, Yt)tez+ is constructed as follows: if there is a birth at 
time t (i.e. if Z^+i > Z^), the types of the new individuals are successively 
picked at random in the population; if there is a death at time t (i.e. if Z^+i < 
Zt)^ the types of the killed individuals are successively picked at random in 
the population; finally, if Zj+i = Zj, then Xj+i = Xt and Yt+i = Yt. 

For example, the transition probability from (i,j) to {i + j + I) for 
(i, j) e and A;, / > 0, A; + Z > 1 is 

fc + A z(z + 1) . . . (z + - 1) j(j + 1) • • • (j + / - 1) 

k ) {i+m+j + i)...{i+j + k+i-i) 

After some algebra, one gets the following formulas for the transition prob- 
abilities: for all / > and /c > such that I + k > 1, the Markov chain 
{Xn,Yn)n>o has transitions from {i,j) G Z,^ to 

{i + kj + 1) w. p. TT (j j+i) := Aj^k^iL Pi+j, i+j+k+i 

\ . k+l ) 

{i-k,j -I) w.p. 7r(jj),(j_fcj_i) := Pi+j, i+j-k-i ^ ^ 

\k+l> 

(ij) W. p. Tr(^i,j),(i,j) ■■= Pi+j,i+j: 

with the convention that (*.) = if ? < 0, j < or j > -i. In particular, once 
one component of the process is 0, it stays zero forever. We denote by 

n := {'^{i,j),{k,i))ii,j),{k,i)ezl 

the transition matrix of the Markov process {X, Y). 

The state space of the Markov chain Z will be denoted by Sz, and the 

state space of (X, Y) by S. We are going to consider two cases: the case 
where Z has finite state space Sz '•= {0, 1, ... , N} for some N > 0, and the 
case where Z has infinite state space Sz '■= Z_|_. In the first case, the state 
space of (X, Y) is the set 

rN:={it,j)eZl:i + j <N}. (37) 
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In the second case, S := Z^. 

We also define the sets S* := S H N^, 7^ := 7]v H and S*z := Sz \ {0}. 
Finally, let Ho be the restriction of the matrix Ho to (i.e. the matrix 
obtained from Ho by suppressing the row and column of index 0) and let 11 
be the restriction of the matrix 11 to 5*. 

Extending the usual definition for Markov chains, we say that a matrix 
M = {'mij)ij^_A is reversible with respect to the measure {^i)i(^A if /ii > 
and fiiTriij = fijiriji for all i,j & A. 

For all > 0, we define 

Vd := {t; e M*^ : = Pd{i,j)ui+j with u G R-^^}, 

where we recall that the polynomials are defined in Theorem 13.11 and 
Pi = P[^\ Note that, by Proposition 13.21 (e) . a vector Vij = Pd{i, j)uij^j G Vd 
is characterized by the values of Uk for k > d only. 

For all d > 0, we also define the matrix 11^ := {pnjn) {n,m)£S , n>d, m>d, where 
for all (n, m) G S such that n > d and m > d, 

( (m+d-l\ 

\ m—n ) •£ ^ 

/^_ix Pn,ra it ITl > U, 

\m—n) 

C-t) ^ (38) 

-T-iT^ Pn,m itm <n, 
\n—m/ 

Pn,n if m = n. 

All these notation, as well as those introduced in the rest of the paper, 
are gathered for convenience in Appendix [XI 

The following result is the basis of all results in this section. 

Proposition 4.1 For all d > 0, the vector space Vd is stable for the matrix 
n. In addition, for all Vij = Pd{i, j)ui^j G Vd, 

Olv)ij = Pdii,j)Oldu)i+j. 
Proof Using fl2Uap inductively, we have 

J2 (^) ^(^+1) • • • {X+k-l)Y{Y+l) . . . {Y+n-k-l)Pd{X+k, Y+n-k) 

k=0 ^ ^ 



p{d) 



{X + Y + d){X + Y + d+l)...{X + Y + d + n- l)Pd{X, Y) 
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for all (i > and n > 1, which can be written as 



k=0 



X + Y + d + n-l 
n 



PdiX,Y) 



for {X,Y) G Zi. Similarly, an inductive use of (120bp yields 



fc=0 



for all d > 0, n > 1 and (X, Y) E Z^, where we recall the convention (°) = 
if a < 0, 6 < or 6 > a. 

Proposition 14.11 then easily follows from these equations and from the 
transition probabilities fl36p . □ 



4.1 The case of finite state space 
4.1.1 Eigenvectors of 11 for finite state spaces 

In the case where Z has finite state space, the main result of this section is 
the following. 

Theorem 4.2 Assume that Sz = Tn for some N > 0. 

(a) For all d > and all right eigenvector {un)neSz,n>d ofUd for the eigen- 

value 6, the vector 

= Pd{iJ)ui+j, (ij) G S (39) 

is a right eigenvector for the eigenvalue 9 of the matrix 11, where the 
polynomials are defined in Theorem \3.1\ and where Pi = P^^K 
In addition, if d > 2, v is also a right eigenvector for the eigenvalue 
of the matrix 11. 

(b) All the right eigenvectors of 11 are of the form ^^) . or possibly a linear 

combination of such eigenvectors in the case of multiple eigenvalues. 
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(c) Assume thatUo admits a positive reversible measure {fin)n&s^- Then, the 
matrix U is reversible w.r.t. the measure 

^(mV=^^^^, V(^,j)g5*, (40) 

and hence is diagonalizable in a basis orthonormal for this measure, 
composed of vectors of the form for d > 2. 
In addition, U is diagonalizable in a basis of eigenvectors of the form 
ford> 0. 

Hence, the right eigenvectors of the transition matrix of a finite N2dMC 
can be decomposed as the product of two terms, one depending on each 
population size, but "universal" in the sense that it does not depend on the 
transitions matrix IIo of Z, and the other depending on the matrix IIo, but 
depending only on the total population size. 



Remark 4.3 There is some redundancy among the right eigenvectors of U 
of the form P{i,j)ui+j for P = Pq, P^-^-* or P{^ : if u is a right eigenvector 
of Hi, the vectors 

(mi+j)(ij)e5 and {jui+j)(^ij)^s 



are right eigenvectors of U for the same eigenvalue. In particular, iui^j + 
jui^j is an eigenvector of U of the form PQ{i, j)u^_^j. This will also be true 
when Z has infinite state space. 

Remark 4.4 In the following proof (and also in the case of infinite state 
space), no specific use is made of the fact that the matrix U is stochastic. 
Therefore, Theorem \4.^ also holds true in the case of a continuous-time 



N2dMC, where the matrix Hq is now the infinitesimal generator of the process 
Zt = Xt + Yt. 



Proof Point (a) is an easy consequence of Proposition 14.11 

For all < < A^, the matrix 11^ is conjugate with its Jordan normal 
form. Let {u^'^'''^}d<k<N denote the basis of corresponding to this 

normal form, where u^'^^'^ = {u^f^'^ ,u[^^f . . . ,u^^^'''). Then, the family of 
vectors 

^ := {(Pd(^,J>!S')(,,)er^. :0<d<N, d<k<N] 
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is composed of (A^ + 1) + + (A^ - 1) + . . . + 1 = (A^ + 1)(A^ + 2)/2 = \Tn\ 
elements. Moreover, one can prove that it is linearly independent as follows: 
since {u^'^^'^}d<k<N is a basis of C^"'^"'"^, it is sufficient to check that 

N 

E^'^(^'^>JS = 0' v(^,j)erA. (41) 

implies that f*^'^) = for all < c? < A^, where v'''^'' = {v^f \ . . . ,v^^) € 
(^N-d+i^ seen as a subspace of C^"*"^ by putting the d first coordinates to 
be zero. Given k < N, the equality (HTj) for i + j = k combined with 
Proposition 13.21 (f) yields t>[°^ = . . . = t>[,'^'' = 0. 

Therefore, J-" is a basis of C^'^ and, by point (a), the matrix 11 has a 
Jordan normal form in this basis. Point (b) is then staightforward. 

If Ho admits a positive reversible measure fi, it is straightforward to check 
that the vector u in (HOj) is a reversible measure for 11, and hence the first 
part of Point (c) is true. 

In addition, the matrix Hi is reversible w.r.t. the measure 

/^n^ := 2nVn, n G >S|, 

which implies that Hi admits a basis of right eigenvectors orthonormal w.r.t. 

Similarly, ITq admits a basis of right eigenvectors orthonormal w.r.t. fi. 
By Point (a), this gives A^+1 (resp. A^) right eigenvectors of IT of the form (139|) 
for (i = (resp. d = 1). Together with the basis of right eigenvectors of 11 
obtained above (extended by zero on {0} x N and N x {0}), this gives a basis 
of eigenvectors of 11 and ends the proof of (c). □ 

4.1.2 Example: 3-colors urn model (or 3-types Moran model) 

The class of transition matrices given in ( 15^ can be obtained by composition 
and linear combinations of the transition matrices II^")'*' = i))){id),{k,i)(^s^ 
Ylin)- ^ (7r{3^(^_;)))(ij),(fe,;)e5 and n(") = i7^(^j^^^k,i)))ii,j),ik,i)es, where for all 
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n > 1 

ii k = i + 1, I = j, i + j = n 
ii k = i, I = j + 1, i + j = n 
otherwise, 

ii k = i — 1, I = j, i + j = n 
ii k = i, I = j — 1, i + j = n 
otherwise, 

ii k = i, I = j, i + j = n 
otherwise. 

One easily checks, first that the vector spaces Vd for all d > are stable 
for all these matrices, and second that, for the matrix (136|) . 

n= 5^ p„,^(n(")+)— " + Pn,n^(n(")-)"-'- ^ p„,„n("). 

Hence, the vector spaces Vd are trivially stable for such matrices. 

One may however recover a much larger class of matrices for which Vd 
are stable vector spaces, by considering the algebra of matrices spanned by 
the matrices n*^"-'^ and n*^"-'. Below, we study in detail such an example. 

Consider an urn with balls of three different colors and consider the 
following process: one picks a ball at random in the urn, notes its color, puts 
it back in the urn, picks another ball in the urn and replaces it by a ball 
of the same color as the first one. The number of balls of each colors then 
forms a Markov chain, which can be viewed as the embedded Markov chain 
of the 3-types Moran model, defined as follows: consider a population of N 
individuals, with 3 different types. For each pair of individuals, at rate 1, the 
second individual is replaced by an individual of the same type as the first 
individual in the pair. 

Let i denote the number of balls of the first color, and j the number of 
balls of the second color. Then, there are N — i — j balls of the third color. 
The transition probabilities of this Markov chain are as follows: given that 



TT 



(n)- 



TT 



(n)- 



and for all n > 



TT 



(n) 
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the current state of the process is the state at the next time step is 



+ with probabihty 

(z — l,j) with probabihty 

+ 1) with probabihty 

— 1) with probabihty 

{i + l,j — 1) with probabihty 

{i — l,j + 1) with probabihty 

{i,j) with probabihty 



i{N-i-j) 
Ar2 , 

i(N-i-j) 
Ar2 , 

j{N-i-j) 
Ar2 , 

j{N-i-j) 

iy2) 
V 



These transition probabihties do not not have the form (136|) . However, a 
variant of Proposition 14.11 and hence of Theorem 14.21 apply to this process, 
because of the fohowing observation: let us construct the matrices n"*", 11", 
n+, n+ from the matrices = (p+^)n,mG5z, = iPn,m)n,mGSz^ = 



iPn,m)n, 



m&Sz • 



{p1 



mjn,m. 



(iSz respectively, exactly as 11 was constructed 



from Ho in (l36B . where 



n,n+l 



k_ 

TV' 



Pn,n 


1 - 


k 


Pn,m 


= otherwise. 


Pn,n 


1 - 


k 


Pn,m 


= otherwise. 


fn,n 


1 - 


k{N-k) 


Pn,m 


= otherwise. 


Pn,n 


1 - 


k{N-k-l) 
N2 ' 


Pn,m 


= otherwise. 



Pn,n—1 AT' 

JM- ^ k{N-k) 
Pn,n+1 ]\f2 5 

+ ^ k(N-k-l) 
Pn^n+l Ar2 ; 

Then the transition matrix of the 3-colors urn model is given by 

n = n+n^ + n+ -fi-. 

In particular, the vector spaces Vd for < (i < are all stable for this 
matrix. 

The transition matrix 11 has absorbing sets {(z,0) : < z < A^}, {(0,i) : 
< i < A^} and {(z, AT - z) : < i < A^}, and absorbing states (0, 0), (A^, 0) 
and (0, A^). The restriction of the matrix 11 on the set 

S** ■.= {{i,j):i>l, j>l, i+j<N-l} 

admits the reversible measure 



1 



IjiN 



J) 



Hence the matrix H admits a family of right eigenvectors null on the absorb- 
ing sets, which forms an orthonormal basis of L^(iS**, u). 
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One easily checks, using ( 15^ . that V(ij) = Pd{i, j)ui+j is a right eigenvec- 
tor of n for the eigenvalue 6 if and only if for all < A; < 

9'uk = {N — k) [{k + d)uk+i — 2kuk + {k — d)uk-i] 

= (N -k) [k{uk+i - 2kuk + Uk-i) + d{uk+i - Uk-i)] , (42) 

where 

^ = ^ + — m • 

Now, the Hahn polynomials Qn{x; a, /3, N) introduced in (133|) satisfy (cf. 
e.g. HU) 

-n{n+a+f3+l)Qd{x) = x{N+/3-x)Qn{x-l) + {N-l-x){a+l+x)Qn{x+l) 

- [x{N + /3-x) + {N-l- x){a + 1 + x)]Q„(x). 

Hence, for all < n < N — d, (H2l) admits the polynomial (in k) solution of 
degree n 

Uk = Qn{k - d]2d-l,-l,N - d+1). 

If n > 1, this polynomial must be divisible by (A^ — A;), so we can define the 
polynomial R^''^\x) of degree — 1 as 

{N - X)i?(f •'=)(X) = Q„(X -d-2d-l,-l,N -d+ 1). 

Obviously, the family of vectors (!,...,!) and {{N—d)R^''''\d), . . . , R^''^\N— 
1), 0) for l<n<N — dis linearly independent and hence forms a basis of 
the vector space M-^"*^"*"^ of real vectors indexed hy d,d + 1, N. In addi- 
tion, (H2II cannot admit any other linearly independent solution and hence, 
necessarily, Rit^''^\k) = ioi all n > N — d and d < k < N — 1. 
We have obtained a basis of right eigenvectors of 11 of the form 



0<d<N-l, l<n<N-d ' 

and the eigenvalue corresponding to the eigenvector Pd{i,j) if n = 0, or 
Pd{i,j){N - z - j)i?(^'")(z + j) if n > 1, is 

n{n - 1) + 2nd - d{d - 1) {d + n){d + n-l) 

Od,n := 1 — = 1 — 



iV2 
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Similarly as in the proof of Proposition 13. 3^ this family of eigenvectors can 
be seen as a singular limit case of those obtained in [T?] for the multitype 
Moran model with mutation or immigration. 

Note that in the case of the 2-colors urn model, one can easily check 
that a basis of right eigenvectors of the corresponding transition matrix is 
given by (1, ... , 1) and {NRi^''^\o), . . . , Ri^''^>{N - 1), 0) for 1 < n < A^. 
Hence the spectrum is the same in the 2- and 3-colors urn models, although 
the multiplicity of each eigenvalue is different. In the case of two colors, 
the eigenvalues have the form 1 — k{k — 1)/N'^ for < k < N, each with 
multiplicity 1 (except for the eigenvalue 1, with multiplicity 2). In the case 
of three colors, the eigenvalue 1 — k{k — 1)/N'^ has multiplicity k+1 (except 
for the eigenvalue 1, which has multiplicity 3). 

Concerning the eigenvectors in lJ{S**, v). they are given by 

{^3{N - ^ - j)Qd(z, "^H^ + j)}2<.<^_i, i<„<^., , 
and for all 3 < < A^, the eigenspace for the eigenvalue 1 — k(k, — l)/iV^ is 

V, := Vect{zj{N - i - j)Q2(^, j)4-2^(^ + j), • • ■ , 

We shall end the study of this example by giving an apparently non-trivial 
relation between the polynomials and Rn''^\ Because of the symmetry of 
the colors, we have 

Vk = Vect{zj(iV - z - J)Q2{^, N-i- j)RfJ\N - j), • • • , 

^J{N - I - 3)QkS, N-i- j)Rf'''-'\N - j)} 
= Vect{zj(iV - z - j)Q2{N - I - j,j)RZf{N - . . . , 

ij{N - t - 3)Qk-i{N - % - j,j)r[''^'-'\n - t)}, 

and hence 

Vect{g2(^, j)^!"!'?!^ + j), • • . ,Q„-i(^, + 

= Vect{Q2(z, N-z- 3)R^n-l\N - j), . . . , Q„-i(2, N-i- j)M'^'"~'^(iV - 3)}{i,ms» 
= Vect{g2(iV - I - 3,3)R^n-l\N - 0, ... , Q„_i(iV - i - j, j)Rf'''~'\N - 
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4.2 The case of infinite state space 



The goal of this subsection is to extend Theorem 14. 21 to the case where Z has 
infinite state space and the matrix Ho is a compact operator and admits a 
reversible measure. To this aim, we need first some approximation properties 
of n by finite rank operators. 



4.2.1 Approximation properties 

Recall that Ho is a Markov kernel on Z+ absorbed at and that Ho denotes 
its restriction to M (equivalently. Ho is the sub-Markovian kernel on N cor- 
responding to Ho with a Dirichlet condition at 0). We assume that Ho is 
reversible with respect to a positive measure /i on N (not necessarily finite). 
For any G N, consider the sub-Markovian kernel IIq^^ on N defined by 



Uo{x,y) if x,y<N 
otherwise. 



In other words, IIq^'' = PrArlloPrAr, where PrTv is the projection operator 
defined by PrAr(Mi, U2, ■ ■ ■) = ("Ui, • • • , Un, 0, . . .). 

The kernel Hq^'' is not Markovian (since n[,^)(x,N) = Ofor X > N) , but it 
can be seen as the restriction to N of a unique Markovian kernel nQ^"* on 
absorbed at 0. With this interpretation, we can construct the matrix 11*^^^ 
from IIq^'* exactly as the matrix 11 was constructed from Ho in the beginning 
of Section H _ 

Of course IIq ^ remains reversible with respect to yU, and thus, like Ho, 
it can be extended into a self-adjoint operator on L^(N, /x). We denote by 
III ■ llo the natural operator norm on the set of bounded operators on L^(N, /i), 
namely if K is such an operator, 



III X III := sup 

«eL2(N,/.)\{0} ll^ll^ 

If furthermore K is self-adjoint, we have, via spectral calculus, 



II llo := sup 



\{u,Ku)f,\ 

|2 



nGL2(N,/i)\{0} W^llfi 

The next result gives a simple compactness criterion for IIq. 
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Lemma 4.5 The operator Uq acting on L^(N, //) is compact if and only if 

lim |||no-n(^^|||o = o. 

N—^oo 

Proof Since for any N G Z_|_, Ifg^'' has finite range, if the above convergence 
holds, IIo can be strongly approximated by finite range operators and it is 
well-known that this implies that ITq is compact. 

The converse implication can be proved adapting a standard argument 
for compact operators: assume that Ho is compact and let e > be fixed. 
Then no(-B) is compact, where B is the closed unit ball of L^(N, //). Hence 

n 

1=1 

for some n < +oo and ■'tp'^^\ . . . /tp^"'^ G no(-B), where B{^Jj,e) is the closed 
ball centered at ip with radius e. For any i < n, since ip^'^^ € L^(N, /x), there 
exists Ni such that ^k>Nii'^k^)'^l^k < and thus 

n 

1=1 

In other words, for all (p E B, there exists i < n such that \\Ilo^p—PrN^^/J^''^ ||^ < 
2s. This implies that ^ ^ 

||no<^-Pr;vno<^IU < 26, 

where = snp{Ni, . . . , N^}, i.e. |||ffo - Privffr^|||o < 2£. Since ff^^^ = 
PrjvIIoPr, we obtain that 

lim |||noPr^-nf^|||o = o. 

In order to complete the proof, it only remains to check that 

lim |||no-noPriv|||o = 0. 

N-^+oo 

If this was false, one could find a sequence (ip^^^) n>i in B such that (p^j^^ — 
for all A; < and llnof/?^"^^ ||^ would not converge to 0. Such a sequence 
{'fi^^^)N>i weakly converges to 0. Now, another usual characterization of 
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compact operators is the fact that the image of weakly converging sub- 
sequences strongly converges to the image of the limit. In other words, 
IIIIoV?^^'' 11^ — 0. This contradiction ends the proof of Lemma [4.51 □ 

The interest of IIq^'' is that it brings us back to the finite situation. Let 
Hq^'* be the restriction of H^^^ to |1,A^], which can be seen as a x 
matrix. We have for instance that the spectrum of Hq^'' is the spectrum of 
Hq^^ plus the eigenvalue 0. 

We are now going to see how the results of Section 14.11 are affected by the 
change from IIo to IIo^''. More generally, we consider two Markov kernels IIo 
and rig on Z+ absorbed at 0, whose restrictions to N, IIo and IIq, are both 
reversible with respect to fi. We associate to them 11 and 11' defined on as 
in and their respective restriction to N^, 11 and 11'. We also define the 
matrices 11^ and 11'^ for > 1, as in (138 p . Note that 11 and 11' are reversible 
with respect to z/, defined in (HOl) and it is straightforward to check that, for 
any d > 1, 11^ and 11'^ are both reversible w.r.t. fi^'^^ = {fJ'if^)neSz,n>d defined 
by 

/ii''^ := 2 n (^^/_\ /in, neN, n > d. (43) 

We will denote ||| ■ ||| and ||| ■ 1^ the operator norms in L^(N^, u) and L^(Nd, fj.^^^), 
where := {d, d+1, . . .}. The next result shows that, if one takes II'q = nQ^"*, 
the approximation of Ho by Hq^'* behaves nicely. 



Proposition 4.6 We always have 

|||n-n'||| = sup iinrf-n'jiu 

d>2 

Furthermore, ifHo — II'q > (in the sense that all the entries of this infinite 
matrix are non-negative), then 

wd > 1, lUd+i - K+il\d+i < llind - n'jid (44) 

and 

|||ni-n;i||i = |||no-n[,|||o. (45) 

In particular, 

|||n - n'l = |||n2 - u'^h < |||no - Kh- 
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Proof For d>2, denote by the set of f G L^(N^, u) of the form 

Vi,jGN, Vij = Pdii, j)ui+j 

with u G L^(Nd, /i^''-'). We denote by v{d,u) the sequence v defined by the 
above r.h.s. The definitions of u, fx^'^^ and Proposition 13.21 (h) enable us to 
see that the mapping 



V) 



is an isometry. 

Proposition 13.21 (h) also shows that and V'^, are orthogonal subspaces 
of L2(N2, z/) for all d,d' >2,d^ d' . We actually have 

V{n\u) = ^V,. (46) 

d>2 

Indeed, let v G L^(N^, v) be orthogonal to for all d>2. For all d>2, we 
define the vector 

The Cauchy-Schwartz inequality and Proposition 13.21 (h) imply that v^'^^ G 
L2(Nd,/i(^)) and since v is orthogonal to V^, by the definition of V^, the vector 
yid.) orthogonal to L^(Nd, /i^'^-'), i.e. v^'^^ = 0. Fixing / > 2 and applying 
Proposition 13.21 (f), one deduces from the equations v^'^^ = for 2 < < / 
that Vi^i-i = foT 1 < i < I — 1, and thus v = 0, ending the proof of ( H6|) . 

Now, Proposition 14.11 show that is stable by 11 and IT and, more 
precisely, 

Vn G h\Nd, /i^'^^), Il[v{d, u)] = v{d, Uau) and Il'[v{d, u)] = v{d, n». 
It then follows from f l46p that 

|||n - n'li = sup iin^ - n;,!!^. 

d>2 

Let us now come to the proof of (144 p . Let c? > 1 be fixed and define 
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I.e. 



and 



^'^ 2d{2d+l) 
j + d 



yt,j>d+l, h!^:={ 



if j > i, 

z — a 
. 1 if « = J- 



j -d 



For any u e lJ{Nd+i, fi'-'^^^''), we get 



\u\ 



< 



\U^\ \U,\ 



\u\ 



Ad) 



sup 



\9j' 



where 



yi>d, Ui := xl gf^ \ui{x)\, 



and Ui — Q ill — d. It is clear that u e L^(Nrf, /x*^'^-'), so, taking the supremum 
over u e L^(Nd+i, /x^'^^^)), we have 



lin.^i - n' 



d+i - < sup 

Hence, it only remains to show that 



. ^/i(l)|||n.-ni 

\9. 



(d) hj 



d - i-i-d\\\d- 



sup . 



id) 
i 

(d) '"hJ — 



1 0*''' (d) 

Since \i ^h]/ = 1 for alH > 1, it is sufficient to prove that 



yj >i>d+l, 



(d) 

id) 'Kj - ^ 
9j 
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since the l.h.s. is symmetrical in For j > i > d + 1, we compute 



< 1, 



ending the proof of (jS]). A similar computation using the fact that 



,(0) and := - "i: 



are given by 



V.,J>1, ^r = 2z^ and = ^ 

leads to (USD . □ 



Again, let H*^^-* be the restriction of li^^^ to Tj^, which can thus be seen as 
a finite x matrix. Similarly to the remark after the proof of Lemma H75| 
the spectrum of 11*^^'' is the spectrum of n^^-' plus the eigenvalue 0. 

4.2.2 Spectral decomposition of infinite, compact, reversible N2dMC 

The following result is an immediate consequence of Lemma 14.51 and Propo- 
sition 14.61 

Corollary 4.7 //IIo is compact and reversible, the same is true for 11. 
We can now extend Theorem 14.21 to the infinite compact, reversible case. 

Theorem 4.8 Assume that Z has an infinite state space, i.e. Sz = 
(a) Theorem\4.^ (a) also holds true in this case. 



(b) If Hq is compact and reversible w.r.t. the measure (/in)neN; then, there 
exists an orthonormal basis o/L^(N^,i/) of right eigenvectors of U of 
the form for d > 2, where v is defined in ( [^^ . Moreover, any right 
eigenvector ofU in L^(N^, u) is a linear combination of eigenvectors of 
the form all corresponding to the same eigenvalue. 
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(c) Under the same assumptions as (b), there exists a family of right eigen- 
vectors of n of the form 

{1} U Pf^^(U)nS'}^^^ U U (47) 

d>2 

which is a basis of the vector space 

V := {v e R^l : V,, = a + -^.J^) + -^vi% + .fj, V., j G 

wzthaeR, v^^\v^^^ eV{N,iJ,) andv^^^ eV{N'^,iy)^ (48) 

in the sense that, for all v & V, there exist unique sequences 
{Pi}i>i, md {'~fdi}d>2, i>i o-nd a unique a G M such that 

^,, = a + ^a,m2/ + ^Aj«;S'+ E 7.zP.(^, jO^S^, (49) 

l>l 1>1 d>2 1>1 

where the series ^lOiiku"^^'^ and Piku"^^'^ both converge for || • ||^ 
(^nd T.d,ndiPd{i,j)u[fj converges for \\ ■ 

Example 1 Assume that Zn = Xn + Yn is a birth and death process, i.e. that 
the matrix Uq is tridiagonal. Assume moreover that all the entries just above 
or below the diagonal are positive (except of course for Pq i, which is since 
Po,o = 1/ It is well-known in this case that there always exists a reversible 
measure fi for Uq. A well-known sufficient condition for the compactness of 
Uq is the case where this operator is Hilbert- Schmidt, which translates in our 
reversible, discrete case as 

E Pi,jPj,i < oo. (50) 

For a birth and death process, letting p^ (resp. qk) denote the birth (resp. 
death) probability in state k and := 1 — — > 0, this gives 

^ {rf+piq^+^ < oo. 

i>0 

As a side remark, note that U is not necessarily Hilbert- Schmidt when Uq is, 
as the condition j) {k i)'^(h3),{k,i)'^(k,i),(ij) < oo is not equivalent to 
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Proof Point (a) can be proved exactly the same way as Theorem 14.21 (a). 

The fact that compact selfadjoint operators admit an orthonormal basis 
of eigenvectors is classical To prove (b), we only have to check that all these 
eigenvectors can be taken of the form (139|) for d > 2. This easily follows from 
the fact that 11^ is compact selfadjoint in L^(Nrf, fi^'^^) for all d > 2, from (H6|) 
and from Point (a). 

The proof of (c) is similar to the proof of Theorem 12.51 (b). Fix v E V 
and define a, v^^\ v^'^^ and v^^^ as in (HSj) . Since /i^^-* = k'^j2k/2, {v^j^^ /k) e 
L^(N, /i*-"'^^) and there exists (q;;);>i such that 



- 5^ aiu^^^'' = J2 c^ikuf^'', Vfc G S*z, (51) 
l>i i>i 



where the convergence of the first series holds for || ■ ||^{i), and thus of the 
second series for || ■ A similar decomposition for f and the use of (b) 
for v^^^ complete the proof of ( H9|) . 

It only remains to observe that, for all f G V, the equation 



, ^ (1) , J (2) , (3) 



uniquely characterizes a G M, v^''\ v^'^^ G L^(N, /i) and v'^^^ G L^(N^,z/ 

(3) (3) 



Indeed, since fg^- = f^Q* = 0, one must have a = vqa, v^^^ = Vifl — a and 



(2) 1-1 

Vj = f oj — a. U 



5 On Dirichlet eigenvalues in the complement 
of triangular subdomains 

In this section, we consider the same model as in the previous section, and 
we assume either that S = Tn is finite or that S = and the restriction Ho 
of Ho to N is compact and reversible w.r.t. some measure /i. We recall that 
rk = eZl:i + j < k} and = TkON^ and we define S*^ := S*\%U 

for all k>2. Notejhat = S*. We also define SI := 5\({0} xZ+). Finally, 
for A; > 0, we call 11^ the restriction of the matrix Uq to {i E Sz '■ i > k}, and 
for k > 1, Uk the restriction of the matrix 11 to S^. Note that this notation 
is consistent with the previous definition of IIq and that 112 = n. Again, all 
the notations of this section are gathered for reference in Appendix [XI 
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5.1 The case of finite state space 

Let us first assume that S = Tn ioT some > 1. 

For 1 < A; < A^, the Dirichlet eigenvalue problem for the matrix 11 in the 
set SI consists in finding 6* G C and v in R^, such that 

(nt;)(ij) = Ov^ij) V(z, j) e SI 
t;(„) = W{z,j)eS\Sl. 

This is equivalent to finding a right eigenvector of 11^ and extending this 
vector by to indices in S\Sl. For all A; > 1, we define as the supremum 
of the moduli of all Dirichlet eigenvalues in S^.. By Perron- Frobenius' theory, 
is a Dirichlet eigenvalue in SI- 

For all d > 0, we also define 9^*^^ as the supremum of the moduli of all 
eigenvalues of 11 corresponding to right eigenvectors of the form Pd{i, j)ui+j. 
Again, by Theorem 14.21 (a) and Perron- Frobenius' theory, 9^^^ is an eigenvec- 
tor of n of the form P(i{i, j)ui^j. Note that, for all d > 2, because of Propo- 
sitioning] (c) and (e), any right eigenvector of 11 of the form Pii{i, j)ui^j is a 
Dirichlet eigenvector of 11 in S^, for all 1 < d' < d. In particular, 9jj', > 
The next result gives other inequalities concerning these eigenvealues. 

Theorem 5.1 Assume that S = Tn- 

(a) Then, 

9i > 92 > 9^ > . . . > 9^_^ > 9^ 
II II VI ... VI II 

1 = 0(0) > ^^(1) > 0(2) > 0(3) >...> QiN-l) > Q{N)^p^^^_ 

(b) IfUk-i is irreducible for some 1 < k < N — 1, then 

Ok > Ok+i, ^^'^ > ^^'^'^ (53) 

and 

9^>9^^\ ifk>3. (54) 

If Ho is irreducible and > for some I <i < N, then 9^^^ < 1. 
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Proof Since the matrix Ho is stochastic, it is clear from Theorem 14.21 (a) 

that = 1. By ([2S]), p'k)m = f Pn,m for all 1 < n, m < and thus Uqu = Ou 
if and only if IIii; = dv, where f„ = Un/n. Hence, the largest eigenvalue of 
Ho is e^^\ Since 

'I 



n, 







V n, 



for some (column) vector v G M;!]^, it is clear that 6^^^ < 9'^^\ Ordering 
conveniently the states in SI, the matrix IIi has the form 

for some rectangular nonnegative matrix Q, since the set {0, 1, ... , A^} x {0} is 
absorbing for the Markov chain. Again, since this matrix is block triangular, 
we have 9^ = maxj^*-^-*, ^^}. Since 112 = H, Theorem 14.21 (b) shows that 

k>2 

Since in addition Un = pN,N^d and IItv = Pn,n, Theorem 15.11 (a) will hold 
true if we prove that the sequences {9''''^)i<k<N and {9^)2<k<N are both non- 
increasing. 

By Perron-Frobenius' characterization of the spectral radius of nonnega- 
tive matrices (cf. e.g. [H]), for all 1 < A; < A^, 

V (fc) 
9^'^= sup .^^Um,P^^^ 



ugR'^fe,u>0,M7^0 



where, by convention, the fraction in the r.h.s. is +oo if Ui = 0. Using 
the notation of the proof of Proposition 14. 6^ for all 1 < < A^ — 1 and 
u e R+'+' \ {0}, we have 



.^^ Z^jgN.+i Pi,j ^ .^^ l^j&%+,Pi,jUj\/ 9i /9j l^ij 

~ iGNfc Ui 
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Taking the supremum over u G R.,.''"''^ \ {0} yields 

Qik+i) < 0(k)^ ^57^ 

For all k > 2, the Dirichlet eigenvectors in belong to the vector space 
Uk := {v elSJ''' : V = on Tn\ S^}. 
By Perron- Frobenius' theory again, for all /c > 2, 

0^ = sup ml . (58) 

Since SI C S*^^-^, for all 2 < A; < iV - 1, 

^k+i = sup mf < fc'fc . (59) 

This ends the proof of (a). 

In the case where Hk-i is irreducible for some 2<fc<A^ — 1, itis clear 
that Ilfc and 11^ are both irreducible. Then, by Perron-Frobenius' theory, 
Oj^ (resp. 9^'^^) is an eigenvalue of Ilk (resp. Uk) with multiplicity one, and 
the corresponding nonnegative eigenvector has all its entries positive. In 
addition, 9^ (resp. 9^^^) is the only eigenvalue of 11^ (resp. 11^) corresponding 
to a positive eigenvector. In particular, the supremum in (158|) (resp. (156|) ) is 
attained only at vectors w E yC^"!) having all coordinates corresponding to 
states in Sl_^ positive (resp. at vectors u G (0, oo)^*^). Hence, the inequalities 
in (15^ and (157|) are strict. 

In the case where Hq is irreducible, the same argument shows that 9^^^ > 
9^^\ and since 9^^^ = 9^ and ^(2) = q^, we also have 9^ > 9^. This ends the 
proof of f l53|) . 

In the case where Ilfc-i is irreducible for 3 < /c < — 1, let {ui)i^fif, be 
a positive right eigenvector of 11^. Then the vector Pk{i, j)ui^j belongs to 
Uk and its restriction to is a right eigenvector of 11^. However, its has 
positive and negative coordinates by Proposition 13.21 (g). Therefore (15^ is 
proved. 

Finally, if Hq is irreducible and > for some 1 < i < N, then the 
absorbing state is accessible by the Markov chain from any initial state 
(possibly after several steps). It is then standard to prove that there exists 
n such that the sums of the entries of each line of (Hq)" is strictly less than 
1. This proves that (Hq)" cannot have 1 as eigenvalue, and thus 9^^^ < 1. □ 
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5.2 The case of infinite state space 

Our goal here is to extend the previous result to the case where S = 
and Ho is compact reversible. So let us assume that Ho is compact and is 
reversible w.r.t. some measure fi. 

In the case of infinite state space, when > 2 the Dirichlet eigenvalue 
problem for 11 in consists in finding ^ G C and v G L^(Z^,z/) (where 
the measure u is extended by convention by zero on Z+ x {0} U {0} x Z+) 
satisfying (152|) . Defining the vector space where Dirichlet eigenvectors are to 
be found 

Uk := {v G V{Zl, z/) : f = on Z^ \ 5^}, 
the supremum of the moduli of all Dirichlet eigenvalues in is given by 



Sk = sup 



(60) 



In view of Theorem 14.81 (c), the natural space to define the Dirichlet 
eigenvalue problem in is 

Wi := \v eR^+:v = in {0} x Z+ and Vij = -^t^E + vf^^ V(z, j) G 5*, 
L ^ "I" J 

where t;^^^ G L^(N,/i) and v^^^ G L2(N^z/)}, 

equipped with the norm || ■ ||^^, where \\v\\lj^ = H'W*-"'^^ ||^ + (this norm 

is well-defined since v^^^ and f'-^^ are uniquely defined from v G Ui). Then, 
the Dirichlet eigenvalue problem in consists in finding 6 & C and f G Wi 
satisfying (152|) . We also define 6^ as the supremum of the moduli of all 
Dirichlet eigenvalues in 5*. 

For all d > 1, we also define 9^^^ as the supremum of the moduli of all 
eigenvalues of 11 corresponding to right eigenvectors of the form Pd{i,j)ui^j 
with u G L2(Nrf,/i('^)). By Theorem |M] (a), 

,„p I I . Vd>0. 

«GL2(Nd,M('*))\{0} lpL(d) 



In addition, using the notation defined in the proof of Proposition 14. 6[ it 
follows from (HHl) that 



Qid) 



sup 



,2 ^^>2. 



46 



Comparing this with (1^U|) , we again deduce from Proposition 13.21 (c) and (e) 
that ejj! > e^'^^ for all 2 < d' < d. 

Finally, since the matrix Ho is not reversible, we need to define 9^^^ in a 
slightly different way: 0(0) is the supremum of the moduli of all eigenvalues 
of n corresponding to right eigenvectors of the form Pd{i,j)ui+j with u = 
al + V, where a G M, 1 is the vector of with all coordinates equal to 1, 
V G L^(N, /i) with the convention Vq = 0. 

Theorem 5.2 Assume that S = and that Uq is compact and reversible 
w.r.t. a positive measure fi. Then, for all d>l, 6^ is a Dirichlet eigenvalue 
of n in the set and 6*^'^) is a right eigenvalue of H for an eigenvector of 
the form Pci{i, j)ui+j with u G L^(Nrf, /i^'^-'). In addition, 

01 > 02 > 03 > 0A >■■■ 
II II VI VI 

1 = 0(0) > ^(1) > ^(2) > ^(3) > 0{4) ^ 

Proof For all A; > 1, the fact that 0^=) is an eigenvalue of 11 for a right 
eigenvector of the form Pk{i, j)ui+j with u G L^(Nfc, /iC^-*) follows from Theo- 
rem l4.8l (a) and from the fact that 11^ is compact reversible. Indeed, 111^11^ is 
an eigenvalue of the compact reversible matrix (the corresponding eigen- 
vector can be obtained for example as the limit of nonnegative eigenvectors 
of finite dimensional projections of 11^). The result follows since, necessarily, 

l^^'^l < iin^ik-. 

For all A; > 2, we define U^'^^ as the matrix whose restriction to SI is 
Ilfc and with all other coordinates equal to zero. As the projection of a 
compact operator, this matrix is compact. Since it is trivially reversible for 
the measure u, Theorem 14.81 (b) applies to IlC^). Then, Proposition 14.61 tells 
us that 0j^ = |||n2'^''|||2, where U'^^ is defined from IlC^) as 112 has been defined 
from n. Therefore, the fact that 0l^ is a Dirichlet eigenvalue for 11 in can 
be deduced exactly as above. 

Recall the definition of n(^-* in Section 14.2.11 For any G N, replacing 
n by n(^), we define similarly as above the quantities 0^'^^^ and 0^^'^\ Due 
to the remark after the proof of Proposition 14. 6[ we are again brought back 
to the finite framework. The following result is an immediate consequence of 
Lemma [4.51 and Proposition 14.61 
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Lemma 5.3 We have 



Mk > 2, lim = 

N-^OD 

Vfc > 1, lim e^'''^^ = 

From this lemma and Theorem 15.11 immediately follow all the inequalities in 
Theorem 15.21 that concern ioT k>2 and 9^^^ for k > 1. 

As in the finite case, we easily deduce from the facts that p^/^j = j-pij and 

fif'^ = 2i^/ij that IIoM = Ou with u G L^(N, /i) iff Ilif = 6v with Vi = Ui/i 
and V G L2(N, /i^^^). Therefore, 

^« = sup 

ueL2(N,^j)\{0} II^IU 

Since for all a G M and v G L^(N, /i), no(cil +v) = al + Hof , we deduce that 
6*^°) = sup{l; ^^^)}. Since IIo is substochastic and reversible w.r.t. /i, we have 
for all u G L2(N,/i) 



This yields 

= 1 > 

In order to complete the proof, it only remains to check that 9^ = 9^^^ 
and that 9f is a Dirichlet eigenvalue in 5*. As in the finite case, IIi has 
the block-triangular form (155]) . Therefore, we obviously have > In 
addition, any Dirichlet eigenvalue in which corresponds to an eigenvector 
in Ui which is nonzero on the set of indices N x {0}, must be an eigenvalue 
of Ho corresponding to a right eigenvector in L^(N, Now, if m G L^(N, /i) 
satisfies IIqu = 9^^^u, then Ilif = 9'^^^v with Vi = Ui/i and it follows from 
Theorem 14.81 (a) that the vector -r^Ui^j is a right eigenvector of Hi. Since 
this vector obviously belongs to Ui, we obtain that 9f = 9^^^ and that 9^ is 
a Dirichlet eigenvalue in 5^. □ 
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6 Application to quasi-stationarity in nearly neu- 
tral finite absorbed two-dimensional Markov 
chains 

In this section, we restrict ourselves to the finite state space case for simphc- 
ity: let 5 = 7jv for some N & N. The first coordinate will be referred to as 
type 1 and the second coordinate as type 2. Recall that the sets Z+ x {0}, 
{0} X Z+ and {(0, 0)} are absorbing for the N2dMC considered above, which 
means that each sub-population in the model can go extinct. This means 
that the transition matrix 11 has the form 



after ordering the states as (0, 0) first. Ordering the states in 5 \ {0} as 
{1, . . . , A^} X {0} first and {0} x {1, . . . , A^} second, the matrix Q has the 
form 



where Qj (1 < ? < 3) are square matrices and Ri {1 < i < 2) rectangular 
matrices. 

In this section, we study the problem of quasi-stationary distributions 
(QSD) and quasi-limiting distributions (QLD, see the introduction) for Markov 
processes, not necessarily neutral, whose transition matrix has the form (16T1 - 
|62|) . The classical case [6j for such a study is the case when Q is irreducible, 
which does not hold here. A general result is proved in Subsection 16.11 Our 
results of Section [5] are then applied in Subsection 16.21 to study the quasi- 
limiting distribution of nearly neutral two-dimensional Markov chains. 

6.1 Yaglom limit for general absorbing two-dimensional 
Markov chains 

Let (X„, Yn, n > 0) be a Markov chain on S = Tn, with transition matrix 
of the form f l6T| - [62|) . We do not assume that this process is neutral. We call 
such processes A2dMC for "absorbed two-dimensional Markov chains". 

Under the assumption that the matrices Qi, Q2 and Q3 are irreducible 
and aperiodic, Perron-Frobenius' theory ensures the existence of a unique 




(61) 





Q2 
R2 Q3 



(62) 
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eigenvalue 6i with maximal modulus for Qi, which is real and has multiplicity 
one. Moreover, Qi admits unique (strictly) positive right and left eigenvectors 
for 6i, Ui and Vi respectively, normalised as Ujl = 1 and ViUi = 1, where 
we use the convention that Ui is a row vector and Vi is a column vector, 
and where 1 denotes the column vector with adequate number of entries, 
all equal to 1. In the following result, we use the classical identification 
of column vectors and measures: for example, Vi = ((f i)j)(j,o)e5\{{o,o)} is 
identified with Zl(i,o)6<s\{(o,o)}(^i)*'^i' ^3 = ((^3)(ij)){i,i)G5* is identified 
with E(ij)e5nN2(^3)(*j)%j)- 

With this notation, using the results of Darroch and Seneta Vi ^ 6q 
and 6q ® f2 are trivial QSDs for the Markov chain {X, Y). 

Theorem 6.1 Assume that the matrices Qi, Q2 and are irreducible and 
aperiodic, Ri ^ and R2 0- Then, for any i > 1 such that {i, 0) G S, 

lim £(i,o) F„) I (X„, F„) ^ (0, 0)] = ® 5o, (63) 

n— >+cx3 

and similarly for the initial state (0,i) G S, where C(^ij) denotes the law of 
the Markov chain (X„, y„)„>o with initial consition {i,j). 
Moreover, for any {i,j) G S* , 

lim £(„-)[(X„,r„) I (X„,F„) ^ (0,0)] 

f 1 ® 6q if 9i > 63, and 61 > 6*2, 

60 f 2 if 62 > 9s and 62 > 9i, 

.. + -«^»^o + ^o««.. ^^0,>0,,0,, (64) 

1 + Wil + W2I 

Pij vi ^ 60 + {1 - pij) 60 (g) V2 if 6*1 = 6*2 > 6*3, 
Vi (g) 60 + {1 - q) 60 V2 if 61 = 62 = 63- 



where 



w, = vM6sI-Q^)-', z = l,2, (65) 

^ ^{i,j)i6il - Qs)'^ Riui , . 

~ %,,)(ei/ - Qs)-' {R,u, + R2U2) ^ ' 

and q = — - — ^^"^^^^ ^_ (57") 

V3iRiUi + R2U2) ^ ' 
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To give an interpretation of this result, we will say that there is extinc- 
tion of type i conditionally on non-extinction if the QLD (1641) gives mass 
to all states with positive i-th coordinate. Conversely, we say that there is 
coexistence conditionally on non-extinction if the QLD gives positive 
mass to the set S*. We also say that type 1 is stronger than type 2 (or type 
2 is weaker than type 1) if 6*1 > and conversely if 62 > Oi- 

Theorem 16.11 says that the limit behaviour of the population conditionally 
on non-extinction essentially depends on whether the largest eigenvalue of Q 
is 6*1, 02 or 6*3. If either 6*1 > 62 and 6*1 > 63 or 6*2 > 9i and 62 > O3, the QLD 
is the same as if there were no individual of the weaker type in the initial 
population, and there is extinction of the weaker type conditionally on non- 
extinction. If 63 > 61,62, there is coexistence of both types conditionally 
on non-extinction. Finally, when 61 = 62, both types can survive under the 
QLD, so none of the types go extinct (according to the previous terminology), 
but there is no coexistence, as one (random) type eventually goes extinct. 
Observe also that the case 61 = 62 > 63 is the only one where the QLD 
depends on the initial condition. 

Note also that, in the case where ^3 < max{6'i, 62} and 61 ^ 62, the QLD 
does not depend on any further information about the matrix Q3. In other 
words, if one knows a priori that 63 < max{^i,^2} and 61 ^ 62, the precise 
transition probabilities of the Markov chain from any state in S* have no 
influence on the QLD. The QLD is only determined by the monotype chains 
of types 1 and 2. 

Our next result says that, for any values of 61, 62 and ^^3, all the QSDs of 
the Markov chain are those given in the r.h.s. of when they exist and 
are nonnegative. 

Proposition 6.2 Under the same assumptions and notation as in Theo- 
rem \6. 1\ the set of QSDs of the Markov chain is composed of the probability 
measures p Vi ^ 6q + {1 — p) Sq C^!) V2 for all p E [0,1], with the additional QSD 

V3 + Wi ^ 60 + 60 ^ W2 , . 

I + W1I+W2I 

in the case where 63 > max{0i,02}- 

Proof The fact that all the QSDs giving no mass to the set S* are of the 
{oTmpvi^6Q + {l—p)6Q^V2foT some p G [0,1] is an immediate consequence 
of the facts that the sets {1,...,A^} x {0} and {0} x {1,...,A^} do not 
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communicate and the only QSD of an irreducible and aperiodic Markov chain 
on a finite set is given by the only positive normalized left eigenvector of the 
transition matrix of the Markov chain (cf. f6]). 

Assume now that /i is a QSD for the Markov chain (X„, Yn)n>o such that 
> 0, and write fi = (yUi, /i2, /is), where /ii (resp. /i2, resp. /is) is the 
restriction of fi to the set {1, . . . , A^} x {0} (resp. {0} x {1, . . . , N}, resp. 
iS*). The equation fiQ = 9Q for some ^ > 0, which characterizes QSDs, 
implies that /is is a nonnegative left eigenvector for Q^. Thus, by the Perron- 
Frobenius theorem, /is = afs for some a > and 9 = 9^. Using again the 
formula iiQ = 93Q, one necessarily has 

IJi{9M - Q,) = av^Ri, t = l,2. (69) 

In the case where 9^ > max{^i, 92}, the matrices ^ald — Qi and ^ald — Qi 
are invertible, as shown in Lemma [6.31 below. Thus /i is given by (I68p . 

In the case where 9^ < 9i for i = 1 or 2, we deduce from (I69p that 
(^3 — 9i)fiiUi = avsRiUi. This is impossible since the l.h.s. of this formula is 
non-positive and the r.h.s. is positive as i?i 7^ 0, W3 > and Ui > 0. □ 



Proof of Theorem 16.11 For all {k,l) E S \ {(0,0)}, we want to compute 
the limit of 



P(,,) [(X„, r„) = (k, I) I (X„, F„) ^ (0, 0)] = ^^^^^^ (70) 



as n — )■ +00, where <5("j) (^z) denotes the element of on the line corre- 
sponding to state € S and the column corresponding to state [k, I) G S. 

Therefore, we need to analyse the behaviour of as n — )■ +00. We have 
by induction 

/ \ 

= (71) 

where 

n-l 

Rt^ = J2Q';RiQr'-', ^ = 1,2. (72) 

k=0 
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By the Perron- Frobenius Theorem (see e.g. 

Qr = ^>^^^. + 0((M"1) (73) 

for some a < 1 as — )■ +00, where 1 denotes the square matrix of appropriate 
dimension, whose entries are all equal to 1. We need the following result. Its 
proof is postponed at the end of the subsection. 

Lemma 6.3 // 9i > 63, the matrix Oild — is invertible, its inverse has 
positive entries and 

r'^^ ~ 9^(6 Jd - Q^y'RiUiv, (74) 

as n ^ +00. If 9i < 9^, the matrix 9zld — Qi is invertible, its inverse has 
positive entries and 

R^"^ ~ 9^usVsR,i9sId - Q,)-' (75) 

as n ^ +00. If 9i = 9^, as n +00, 

i?!"^ ~ ne'l-^u^v^RiUiVi. (76) 

Theorem 16.11 can be proved from this result as follows. Let D denote the 
denominator of ([70]). If ^1, ^2 > ^3, dZi) and ^ yield for all (i, j) G S* 

D ^ 9^,5i^,j){9M-Qzr'RiUiv^^+9:^hj){9M-Q^V^ 

as n ^ +00. In the case when 9i > 92, since (^ild — Qs)"^ has positive 
entries, we have D ~ ^"5(jj)(^ild — Q3)~^RiUi. The limit of (170|) when 
n — )■ +00 then follows from f l74p . The case ^2 > ^1 is treated similarly. In 
the case when 9i = 92, 

D ~ ^1 %i) [(^ild - Qsr'RiUi + (^^2ld - Q3)-'R2U2] , 

and the fourth line of follows from Lemma 16.31 
In the case when 9^ > 9i,92, we obtain 

D ~ 0^6^ij)usvs [Ri{9sld - Q^y'l + R2{9M - Q2)-'l + l] , 

which implies the third line of (IMI) . 

Similarly, it follows from (173|) and Lemma 16.31 that 

D ~ 9^6^ij){9ild - Qsy'RiUi if > > 92, 
D ~ n^^-^(5(ij)M3t;3i?iMi if 9i = 9^>92, 

D ~ n^r^(5(,j)M3^^3(^iMi + R2U2) if 9^ = 92 = 9^. 

The proof is easily completed in each of these cases. □ 
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Proof of Lemma 16.31 Assume first that 6i > 6^. One easily checks that 



n-l 



hid - Q,) J2 (^I'Qs = W - ^r+'Q3 • (77) 



fc=0 



Because of f l73|) . the series in the previous equation converges when n +oo. 
Therefore, ^^ild — is invertible and 



n>0 



which has positive entries since Q3 is irreducible. Therefore, it follows from ([73 
and ([77D that 



n-l 



i?S") = J2 Q3R1 [9r^-''uivi + 0((eia)"-i-^l 



k=0 

n-l 



^^^(^ild - QzY^RiUiVi + 0{e^l) + O {{6^ + 1 



fc=0 



where we used the fact that a may be increased without loss of generality so 
that ^3 7^ 6ia, in which case 

k=0 ^ 

Since Ri ^ has nonnegative entries and (^ild— Q3)~^ and uiVi have positive 
entries, the matrix (^ild — Q3)~^RiUiVi also has positive entries, and ( 17^ 
follows. The case 63 > 61 can be handled similarly. 
Assume finally that ^1 = ^3- Bv (173]) . 

n-l 



^S"^ = Yl (^3«3t^3 + o((^3«)'i)) Ri {er'-'u.v, + omar-'^-'i)) 

k=0 

= nO'^-^u^vsRium + O i ^i— {iRium + u^vsRa) j + 0(n(a^i)"-il), 
which ends the proof of Lemma 16.31 □ 
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6.2 The nearly neutral case 

Since 11 in (l6T|l is a block triangular matrix, we have 

Sp'(n) = {1} u Sp'(Qi) u Sp'(g2) u Sp'(g3), 

where Sp'(74) denotes the spectrum of the matrix A, where eigenvalues are 
counted with their multiplicity. 

In the case of a N2dMC satisfying the assumptions of Theorem I6.H with 
the notation of Section HI we have Qi = Q2 = Hq. By Theorem 14.21 (b) and 
Remark IO| {l}USp'((5i)USp'((52) is the set of eigenvalues corresponding to 
right eigenvectors of 11 of the form Po{i, j)ui+j and Pi{i, j)ui^j, counted with 
their multiplicity. More precisely, Sp'((5i) corresponds to eigenvectors of the 
form P^^\i , j)ui+j , and Sp'{Q2) to eigenvectors of the form P^\i, j)uij^j. In 
particular, 61 = 62 = 9^^"^ = 9f, with the notation of Theorem 15.11 Moreover, 
since Q3 = H^^), Theorem O shows that 6*3 = 6^(2) = 9^ < 9^ = 62 and 
Sp'(Q3) is the set of eigenvalues corresponding to right eigenvectors of 11 of 
the form P(i{i, j)ui^j for d>2, counted with their multiplicity. 

In other words, with the terminology defined after Theorem 16. ![ coexis- 
tence is impossible in the neutral case. Since the eigenvalues of Qi, Q2 
and Q3 depend continuously on the entries of these matrices, we deduce that 
coexistence is impossible in the neighborhood of neutrality: 

Corollary 6.4 Let U be the transition matrix of some fixed N2dMC in Tn 
such that Ho and Hi are both irreducible and there exists i G {1,...,A^} 
such that Pi > 0. For any A2dMC {X,Y) in Tn with transition matrix TV 
sufficiently close to 11, coexistence is impossible in the QLD of {X,Y). Let 
9'i,9'2, 9'^ denote the eigenvalues ^1, ^2, ^3 of Theorem \6.1\ corresponding to the 
matrix 11'. If 9'^ ^ 9^, the QLD of {X,Y) is the trivial QSD corresponding 
to the stronger type: if 9[ > 9'2, the QLD of (X, Y) is v[ ® 5q, where v'^ is the 
QLD of (X, 0), and if 9'^ > 9[, the QLD of (X, Y) is Sq ® v'2, where v'2 is the 
QLDofiO,Y). 

A Notations 

We gather here all the notations used at several places in the paper. Most 
of these notations are introduced in Sections H] and |5l 
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A.l General definitions 



• For any measurable subset F of R and any a-finite positive measure 
yU on F, L^(F,/i) is tlie set of Borel functions / on F defined up to 
/i-negligible set such that Jp /^dyU < +00. We denote by {■,-)fj, the 
canonical inner product on L^(F,/i) and || ■ ||^ the associated norm. In 
the case when F is discrete, we make the usual abuse of notation to 
identify the measure /x and the corresponding function on F. 

• For all / X / square matrix M, where / is a finite or denumerable set 
of indices, and for all J C /, we call "restriction of the matrix M to 
J" the matrix obtained from M by removing all rows and columns 
corresponding to indices in / \ J. 

A. 2 Polynomials 

H2{X), H^i^X), . . . are defined in Proposition 12.21 

Po(x,r) = i. 

Pi'\x,Y) = P,{X,Y)=X. 

pf^(x,r) = Y. 

P2{X, Y), PsiX, r), ... are defined in FheoremO 

A. 3 Sets 

Z+ = {0,1,...}. 

N={1,2,...}. 

Nd = {d,d+1,...}. 

Tn = eZl:i+j<N}, where > is fixed below. 

Finite case Infinite case 

5z = {0,l,...,iV}. 
5| = {l,2,...,iV}. 

S = Tn. 
S* = T^. 

SI = eZl:k<t+j< N}, 

for all k>2. 
5* = 7^ n (N X z+). 



SI 

s 

s* 



= z+ 

= N. 



k>2. 

S* = Nx Z^ 



e Zl : k < i + j}, for all 
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A. 4 Matrices 



Ho = {Pn,m)n,m£Sz ^ stochastic matrix such that po,o = 1- 

Ilfc is the restriction of Ho to the set of indices Sz H N^+i, for all A; > 0. 

Pr^v is the projection operator on defined by Prjv(^ii, U2, ■ ■ ■) = {ui, . . . ,Un,0, 



n 



(TV) 


r(^) 



FrjvnoPrAT, in the infinite case (i.e. when Sz = 



IIq is the Markovian kernel on Tj^ whose restriction to N is 11, 



(N) 



IIq^^ is the restriction of IIo^'' to {!,..., N}. 

n = (vr(ij),(fc,/))(ij),(fc,/)g5, where 

\ k+l ) 



IT 



{i,j),(i+k,j+l) 



^i+j+k+l-1 

(:)(]) 



Pi+j, t+j+k+i if (k, I) ezl\ {0}, 



(i+j) 



Pi+j, i+j—k—l 







if (-fc,-/) G Zl, 
otherwise, 



with the convention that Q) = if i < 0, j < or j > i. 
n is the restriction of 11 to 5*. 

Uk is the restriction of the matrix 11 to 5^, for all > 1. 
n*^^^ is constructed from IIq^'* exactly as 11 is defined from Ho. 



n(^) is the restriction of H^^) to T^- 



\Pn,m) (n,m)<=S, n,m 



' fm+d—l 



\ m- 



\m—n) 
t n-d\ 

\n~m) 



Pn,r, 



>cL for all > 0, where for all (n, m) & S, n,m > d, 



ii m > n, 



( " ) 

\n—m/ 
Pn,n 



Pn,r, 



if m < n, 



if m 



n. 



For A2dMC (see Section El]) 

(Qi 

n = ( ^ 1 , where (5= Q2 

\-Ri R2 Qz, 



'I 
r Q 
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A. 5 Measures and vectors 



fi = is a reversible measure for the matrix Ho. 



^ = i^{i,j)){i,j)&s*, where u^ij) 

(d) _ 
(d) 



fi'"' = (/inOneSzHNd for all d > 1, where 



9 



(d) 



2n 
(9f 



n + d — 1 
2d - 1 



/i„, for all n E Sz r\Nd 



for all (i > 0, where 



^ = 2f for all i E S*z and 

Ad) 



{i - d){i + d) 



for all (i > 1 and i G 5^ H Nd+i. 



2d(2d+l) 

h^d) ^ (/i-j )(jj)g5,ij>d+i for all > 0, where 



for all (i, j) G 5* and 



' ] + d 

i + d 
j - d 

i — d 
1 



if j > i, 

if j < i all c? > 1 and (i, j) G iS, j > c? + 1. 

if i = j. 



A. 6 Operator norms (in the infinite, reversible case) 

III" III is the natural operator norm on the set of bounded operators on L^(N, fi). 
Ill ■ III is the natural operator norm on the set of bounded operators on L^(N^, v). 
Ill ■ \d is the natural operator norm on the set of bounded operators on 
L2(Nd,/i(^)), for all d > 1. 



A. 7 Eigenvalues 

We refer to Sections 15.11 and [5.21 for precise definitions in the finite and infinite 
cases. 

6'^ is the biggest Dirichlet eigenvalue of 11 in iS^, for all A; > 1. 

6'^'^) is the biggest eigenvalue of 11 corresponding to right eigenvectors of the 

form P(i{i, j)ui^j, for all c? > 0. 

For A2dMC (see Section EM 
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01, 02 and ^3 are the Perron- Frobenius eigenvalues of Qi, Q2 and Qs, respec- 
tively. 

A. 8 Vector spaces 

Vd := {i; G : Vij = Pd{i,j)ui+j with u G R^^}. 
Finite case 

Wfc = G M'^^ : w = on 7^ \ S*^} for all A; > 1. 
Infinite, reversible case 

U^ = {ve \}{%\, z/) : = on \ SI} for all k>2. 

U^ = \v^R^'^:v,^, = ^v'tl^^vf}, GL2(N,/i), v^^'^ ^\.\n\v)Y 

Vd={v^ L2(N2, v) : = Pd{i,j)ui+j with m G L2(Nrf, /i^'^))} for all d > 2. 
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